3:55 pm

Live Chat Unavailable Tuesday, December 16th, 2am to Wednesday, December 17th, 2am PST

Posted (December 15th, 2014 at 3:55 pm PST) by Brian@dreamhost

Our Live Chat support system will be unavailable this coming Tuesday, December 16th, from 2am to 2am PST (the following morning, 24hours). We will be involving our support staff in our year’s end team meeting. We’ll be taking a company-wide look at the last few months and discuss our performance and improvements moving forward. Our aim is to offer the best service possible and we feel this meeting will only help to improve our Live Chat team’s performance. This was previously announced as being between Tuesday December 16th at 12am till Wednesday, December 17th at 12am, but has been revised to provide the best coverage possible based on our staff’s schedules.

Email support operations will still be available, should you need to get in touch with us. We will resume normal Live Chat operations at 2am PDT the next morning, December 17th. Thank you for your patience and understanding.


2:59 pm

Scheduled DreamCompute Maintenance – December 4th 11am PST (4hrs) (RESOLVED)

Posted (December 2nd, 2014 at 2:59 pm PST) by andrea

On Thursday December 4th from 11AM to 3PM, our cloud engineering team will be investigating a known, intermittent connectivity issue. The team will be working in coordination with one of our networking vendors to diagnose the matter and plan a resolution. We expect network availability to remain consistent, though there is a possibility your instances will be temporarily unavailable during the maintenance window.

We will closely monitor the maintenance and will be posting periodic updates. This maintenance window only affects the DreamCompute service.

If you have any questions or concerns, please don’t hesitate to contact support for assistance.

Update December 4th, 2014, 12:45PM PDT: The maintenance has been completed. We performed some upgrades and first tests show the expected results. We will continue to test, evaluate and see if any further changes are needed in future maintenance periods. If you see any issues with your DreamCompute service after this maintenance please contact support.

Previous updates available after the jump →


2:12 pm

Scheduled DreamObjects Maintenance – December 9th 10am PST (24hrs) (RESOLVED)

Posted (December 2nd, 2014 at 2:12 pm PST) by andrea

As part of our continued efforts to improve our cloud services, on Tuesday December 9th from 10am to Wednesday December 10th 10am, our Systems Engineering team will be performing maintenance on our DreamObjects service. During this maintenance window, you may intermittently experience slower upload and access speeds and some short periods of connectivity loss. Please rest assured that your data is safe and will not be affected.

The purpose of this maintenance is to update the configuration of Ceph, the underlying storage technology powering DreamObjects. The team is making the configuration change to prepare the cluster for additional capacity. The change will trigger the cluster to re-optimize the arrangement of its data and is likely to cause inconsistent availability. The bulk of the optimization will complete within the first few hours and will diminish throughout the day.

We will closely monitor the maintenance and will be posting periodic updates. This maintenance window only affects DreamObjects and the manual account backup feature.

If you have any questions or concerns, please don’t hesitate to contact support for assistance.

Update December 10th, 2014, 10:00AM PDT: The maintenance window is now closed, and the maintenance is complete. We did not detect any issues with the DreamObjects service during the last 24 hours and we are now one step closer to an upgraded and expanded DreamObjects cluster. If you notice any issues since the maintenance please contact support and we would be happy to assist you further. – Justin

Previous updates available after the jump →


3:29 pm

Scheduled DreamObjects Maintenance – December 2nd 10am PST (24hrs) (POSTPONED – see new post above)

Posted (November 25th, 2014 at 3:29 pm PST) by andrea

As part of our continued efforts to improve our cloud services, on Monday December 1st from 10am to Tuesday December 2nd 10am, our Systems Engineering team will be performing maintenance on our DreamObjects service. During this maintenance window, you may intermittently experience slower upload and access speeds and some short periods of connectivity loss. Please rest assured that your data is safe and will not be affected.

The purpose of this maintenance is to update the configuration of Ceph, the underlying storage technology powering DreamObjects. The team is making the configuration change to prepare the cluster for additional capacity. A consequence of this change is an expected rebalance of the data to optimize the storage cluster. The bulk of the rebalance and optimization will complete within the first few hours and will diminish throughout the day.

We will closely monitor the maintenance and will be posting periodic updates.
Please note, this maintenance was originally scheduled for December 1st, 10am PST. However, based upon feedback, we delayed it to the 2nd (Tuesday) to minimize any potential impact for our customers.

If you have any questions or concerns, please don’t hesitate to contact support for assistance.

Update December 2nd, 2014, 2PM PST: Apologies for the late notice: this maintenance has been postponed until next week. There is additional prep work to complete before making the configuration change and it is not yet concluded. We’ll be adding a new status post with the details.


11:54 am

WordPress 4.0.1 – Security Release

Posted (November 20th, 2014 at 11:54 am PST) by Mika E.

WordPress 4.0.1 is now available as a security release for all previous versions of WordPress, and is being applied now to all sites who allow for Background Updates in WordPress, as well as any who use our Auto Update via Panel.

In fact… you may have already been emailed about this by your install.

You can read the release notes for 4.0.1, and this release is coordinated with a Drupal release, as both projects share a potential vulnerability (due to an issue in PHP) that could cause a denial of service attack. The upgrade applies to WordPress versions 4.0.1, 3.9.3, 3.8.5, and 3.7.5.

As a reminder, WordPress itself performs background updates of all minor releases, since WordPress 3.7 was released in 2013. That means even if you have disabled auto updates via panel (which you’d never do, because updating is the best thing since sliced bread), WordPress will still upgrade your 3.7, 3.8, and 3.9 installs to the latest, most secure, version.

If for whatever reason you went and disabled all possible update methods to your site, please log in to your WordPress site now, head to Dashboard → Updates and simply click “Update Now.” It’s just that easy.

Speaking of updates …. WordPress 4.1 is due out in December! Now’s the time to have a look at your sites and make sure all your plugins and themes are up-to-date in preparation for some cool new features. You may even want to upgrade your PHP settings in panel to PHP 5.4, just to be extra cool.


6:36 am

Emergency maintenance on shared web server ‘fullback’ [RESOLVED]

Posted (November 17th, 2014 at 6:36 am PST) by chalise

Our shared web server, ‘fullback’ is in need of an emergency memory replacement, which will be starting shortly.

We expect the maintenance window to last about a half hour and we will be updating this status post with updates as needed.

Please note, this post is only regarding the above upcoming maintenance. If you are having any issues with your hosting service outside the maintenance window, or regarding any other services, please submit a support ticket through DreamHost panel with your details so we can help!

Update – 6:57 am PST: Maintenance is complete! All services on ‘fullback’ are back up and running. Thank you for your patience!


11:16 am

Scheduled DreamCompute Maintenance – November 17th, 6AM PST (30 min) (RESOLVED)

Posted (November 14th, 2014 at 11:16 am PST) by andrea

As part of our continued efforts to improve our cloud services, on Monday, November 17th, from 6am-6:30am PST, our cloud engineering team will be performing maintenance on our DreamCompute service. The maintenance work is minor and is limited to a configuration change for the compute and network component services. During this maintenance window, you may experience short periods of connectivity loss to your instances. We apologize for any inconvenience this may cause.

Update November 17th, 2014, 7:44AM PST: The changes for this maintenance are complete and our testing shows the system working normally. This maintenance is now complete. Should you still run into any issues with DreamCompute, please report it to tech support and we will get it checked out immediately. – Justin

Update November 17th, 2014, 6:30AM PST: Work is continuing on this configuration change, but we do not expect any further disruption in service for customers. We will keep this status post open until all work is complete so that customers are aware and can report issues to tech support if any do arise. Thanks for your patience during this maintenance process. – Justin


11:41 am

Scheduled Network Maintenance for Homiemail-sub5 Email Cluster and subset of Mysql Servers in US-West (Los Angeles) Data Center – November 6th, 10pm PST (4hrs) (RESOLVED)

Posted (November 5th, 2014 at 11:41 am PST) by andrea

We will be conducting maintenance in our US-West (Los Angeles) Data Center starting at 10pm PST on November 6th. The purpose of the maintenance is to upgrade the capacity of the networking hardware to these machines and improve overall redundancy. We are scheduling a window of four hours for thorough pre and post testing, but the actual service interruption will be minimal. We will be closely monitoring the process and will be posting updates as we progress.

Please note, this maintenance affects a subset of mysql servers (see below for the full list) and the mail traffic on homiemail-sub5. No mail will be lost only queued up for later delivery. You can tell if you’re affected by checking for your mysql server and email cluster under the Account Status in your DreamHost panel. We appreciate your patience and understanding while we continue improving our networking system. We will open the comments as soon as we start the maintenance. Until then, if you have any questions or concerns, please let us know by contacting tech support.

The list of affected mysql servers:
straptail
stonefish
stonecat
surgeonfish
stingfish
stickleback
switch
switch
sturgeon
sunfish
surfperch
steelhead
stargazer
squirrelfish
squaretail
squawfish
springfish
squeaker

Update November 7th, 2014, 12:45AM PST: We’re happy to report that maintenance is now complete. If you need any assistance related to this or anything else, please let us know by contacting tech support.

Update November 6th, 2014, 11:38PM PST: We are continuing with the maintenance as scheduled, and all the steps are being completed without impact on services. We appreciate you continued patience, and we’ll post another update within the hour or sooner if there is a status change.

Update November 6th, 2014, 10:38PM PST: So far so good! We will update this post again within the hour or if there is a status change.

Update November 6th, 2014, 10:05PM PST: We have started the maintenance as scheduled. Please keep an eye out for updates as we progress through the steps.


2:01 pm

Upgrading 187 shared web hosting servers to Ubuntu 12.04 Precise – November 8th, 8PM – 2AM PST(RESOLVED)

Posted (November 4th, 2014 at 2:01 pm PST) by Brandon

We’re completing our roll-out of Ubuntu 12.04 Precise to the final 187 shared web servers  this Saturday, November 8th.  While the total estimated maintenance is 6 hours, we expect actual downtime due to the upgrade to be around 45 minutes. A large part of the maintenance window will be spent testing all of the servers post-upgrade to ensure everything is in order. While this upgrade process is expected to go smoothly, there are some things you can do to reduce the chances of additional service interruption. Customers running  Ruby or Python applications should ensure that they are not linking to static versions of any required libraries, as library versions will change. If you’re using a custom PHP.ini file or if you’re unsure, read the instructions on updating it here.   Here is the list of web servers to be upgraded. You can check if you’re one of these web servers by clicking the ‘Account Status’ link within the DreamHost panel and looking for ‘web server’:

abhainndearg
abraham-clark
afzelia
agave
agrajag
allitnil
andy
appling
ardbeg
aten
atum
auchentoshan
balin
balmenach
baratheon
bard
benrinnes
beorn
bernstein
bifur
blackboard
bocote
bofur
bowmore
brunswick
buckshot
bulldog
bungalow
butts
cactus
candler
carter-braxton
catoosa
cavalier
cement
charles-cotesworth
chausson
chavez
checkerboard
checkmate
chesterfield
chrysanthemum
clam
clarke
daily
dancer
daylily
decatur
deckhand
delpuerto
dickenson
dindy
dinwiddie
dodge
dori
dwalin
early
echols
edradour
effingham
evans
fauquier
finley
fireplug
fistfight
foxcraft
franklin
george-ross
george-wythe
glen-elgin
glen-grant
glendronach
glenfiddich
glengoyne
glengyle
glenkinchie
glenlivet
glenlossie
glentauchers
gloucester
goochland
grady
halifax
hancock
hanson
hapi
hart
heard
heimdallr
hodhr
hoenir
horsetail
hovhaness
hugh-williamson
invergordon
isle-of-wight
kininvie
kittyhawk
kneecap
knockdhu
lanier
lumpkin
lyman-hall
maat
macon
madison
mahler
matthew-thornton
meriwether
mignone
miltonduff
mirfak
monroe
montgomery
moonflower
naos
nelson
nihal
nottoway
oin
page
paulding
phact
pierce
poplar
prince-george
prince-william
ptah
pulteney
quitman
ra
rana
reshep
richard-dobbs
richmond
rimsky-korsakov
roanoke
rockbridge
roseisle
russell
sadira
saint-saens
santoro
sawhorse
scapa
scheat
screven
segin
shenandoah
sheoak
signature
skymaster
smelter
snotra
spectator
spotsylvania
springbank
stanford
starlaw
sterope-ii
strathisla
strathmill
stutter
sugarfoot
tamdhu
tazewell
telfair
the-macallan
thomas-heyward-jr
tinkerbell
tool
tracer
transit
trapline
treasure
twiggs
union
vroomfondel
walker
warren
westmoreland
williamson
wise
wonko
york
zem
zwilich
If you have any concerns leading up to the upgrade, please don’t hesitate to contact our support team. They’ll be more than happy to answer any questions you might have! While the upgrade is ongoing, we’ll keep this post up-to-date with new information as it becomes available.
While this upgrade process is expected to go smoothly, there are some things you can do to reduce the chances of additional service interruption. Customers running  Ruby or Python applications should ensure that they are not linking to static versions of any required libraries, as library versions will change. If you’re using a custom PHP.ini file or if you’re unsure, read the instructions on updating it here.
Update November 9th, 2014, 12:50AM PST: All shared web servers have now been upgraded to Ubuntu 12.04 Precise. Thanks for your patience throughout this process!

7:57 pm

Live Chat Unavailable Saturday October 25th 4pm – Sunday October 26th 4pm PDT

Posted (October 23rd, 2014 at 7:57 pm PST) by Brian@dreamhost

Our Live Chat support system will be unavailable this coming Saturday, October 25th, from 4pm to 4pm PDT (the following afternoon). We will be involving our support staff in our annual Halloween party, rewarding them for their excellent service and hard work! We aim to offer the best service possible and we feel our team deserves it!

Email support operations will still be available, should you need to get a hold of us. We will resume normal Live Chat operations at 4pm PDT the next afternoon, Oct 26th. We thank you for your patience and understanding.


 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
5:36 pm

HTTPS Web Site Owners – SSL v.3 Disablement

Posted (October 16th, 2014 at 5:36 pm PST) by Brian@dreamhost

DreamHost has disabled SSL v.3 across our network. After 18 years of being available as an option for encryption & security in browser communications, the SSL 3.0 protocol has finally met it’s demise in the form of a not so cuddly POODLE (http://en.wikipedia.org/wiki/POODLE) attack.

DreamHosters with HTTPS enabled on their web sites are the only customers that will be affected. You can verify if you have SSL enabled on your domain, here. Visitors whose web browsers do not support the more modern TLS protocol will not be able to negotiate a Secure connection.

The good news is those antique browsers are largely retired from use. The most modern browser that will be unable to negotiate a secure connection with SSL v.3’s disablement is Internet Explorer 6.0 with Windows NT 4.0, released Aug 2001. Please don’t hesitate to contact our support team with any further questions in this matter.


8:29 am

DreamObjects Service Connectivity Issues (RESOLVED)

Posted (October 16th, 2014 at 8:29 am PST) by Dan J

Our cloud engineering team is investigating connectivity issues being reported by the DreamObjects service. We do not currently have an estimate as to how long it will take to restore services. All data stored in DreamObjects is safe and we will continue to update this post as we gather more information.

Please note, this does not affect any other DreamHost services, this is specific to DreamObjects.

If you have any questions or concerns, please don’t hesitate to contact support for assistance.

Update October 16th, 2014, 9:50AM PDT:DreamObjects service continues to be working and work continues on backend hardware replacement and improvements. We will continue to monitor our systems for further issues.

Previous updates available after the jump →


12:29 pm

Upgrading 150 shared web hosting servers to Ubuntu 12.04 Precise – October 18th, 8PM – 1AM PDT (RESOLVED)

Posted (October 14th, 2014 at 12:29 pm PST) by Brandon

We’re continuing our roll-out of Ubuntu 12.04 Precise to an additional 150 web servers this Saturday, October 18th. As we’d like to get all of our customers over to this new OS, we will be upgrading 2 batches per week. While the total estimated maintenance is 5 hours, we expect actual downtime due to the upgrade to be around 45 minutes. A large part of the maintenance window will be spent testing all of the servers post-upgrade to ensure everything is in order.

As the Ubuntu image will not support PHP5.2, anyone who still has this version of PHP selected in the panel will be migrated to a Debian server before these web servers are upgraded, and will still have until the November deadline to upgrade to a more recent version of PHP.

While this upgrade process is expected to go smoothly, there are some things you can do to reduce the chances of additional service interruption. Customers running  Ruby or Python applications should ensure that they are not linking to static versions of any required libraries, as library versions will change. If you’re using a custom PHP.ini file or if you’re unsure, read the instructions on updating it here.

 

Here is the list of web servers to be upgraded. You can check if you’re one of these web servers by clicking the ‘Account Status’ link within the DreamHost panel and looking for ‘web server’:

 

aberfeldy
aberlour
aker
alltabhainne
ammit
andriessen
apophis
ardmore
arran
atterberg
atum
auchinblae
auchroisk
bakha
balbair
bat
beech
bennevis
benriach
benromach
bizet
blacktop
bladnoch
blairathol
bliss
brackla
braeval
brian
britten
broadside
bruckner
bruichladdich
buckingham
cameronbridge
carroll
chamomile
charles-city
chatham
cherry
cloudburst
corydalis
cragganmore
craigellachie
dade
dagr
dalmore
dalwhinnie
danielpour
davies
deanston
debussy
deepthought
delling
draeseke
eddie
effingham
fan
fettercairn
fireplug
forseti
franck
geb
gillis
girvan
glascock
glazunov
glen-garioch
glen-moray
glen-scotia
glen-spey
glenallachie
glencadam
glendullan
glenfarclas
glenglassaugh
glenmorangie
glenrothes
glenturret
gloin
gmot
goldmark
guerra-peixe
haralson
hartmann
hathor
heget
hickory
highland-park
hindemith
holmboe
iounn
isle-of-jura
joro
kellaa
khnum
khonsu
kilchoman
king-george
knockando
lalo
linkwood
lochnagar
longmorn
macduff
madison
mafdet
magni
mannochmore
marblewood
melartin
messiaen
mimir
mortlach
mozart
muliphein
nielsen
nighthawk
nystroem
oban
odin
pallando
penderecki
persichetti
porrima
radagast
ras-alhague
rubbra
saga
sarir
saruman
sasin
satis
schubert
shaula
sif
sjofn
springtime
stagecoach
still
strathclyde
strauss
stravinsky
suhail
suk
sumter
tienkwan
tobermory
toch
tournemire
tubin
tullibardine
vivaldi
wagner
wheels
widor
wolfburn

 

If you have any concerns leading up to the upgrade, please don’t hesitate to contact our support team. They’ll be more than happy to answer any questions you might have!
While the upgrade is ongoing, we’ll keep this post up-to-date with new information as it becomes available. If you’d like to know more about the benefits resulting from our switch to Ubuntu, please read this post.

Update October 19th, 2014, 12:18AM PDT: The Ubuntu upgrades have completed and all batches are now back online. The following web servers were not upgraded, and will be upgraded at later date: zem vroomfondel baratheon effingham fireplug.

Update October 18th, 2014, 11:41PM PDT: Batch #1(“aberfeldy” through “cherry”)  has now completed the upgrade to Ubuntu. Batch #2 (“cloudburst” through “goldmark”) has now completed. Batch #3(“guerra-peixe” through “penderecki”) machines are all complete. Batch #4(“persichetti” through “wolfburn”) is complete other than”tienkwan”.

Previous updates available after the jump →


8:14 pm

Upgrading 150 shared web hosting servers to Ubuntu 12.04 Precise – October 12th, 8PM – 1AM PDT (RESOLVED)

Posted (October 7th, 2014 at 8:14 pm PST) by Brandon

We’re continuing our roll-out of Ubuntu 12.04 Precise to an additional 150 web servers this Sunday, October 12th. As we’d like to get all of our customers over to this new OS, we will be upgrading 2 batches per week. While the total estimated maintenance is 5 hours, we expect actual downtime due to the upgrade to be around 45 minutes. A large part of the maintenance window will be spent testing all of the servers post-upgrade to ensure everything is in order.

As the Ubuntu image will not support PHP5.2, anyone who still has this version of PHP selected in the panel will be migrated to a Debian server before these web servers are upgraded, and will still have until the November deadline to upgrade to a more recent version of PHP.

While this upgrade process is expected to go smoothly, there are some things you can do to reduce the chances of additional service interruption. Customers running  Ruby or Python applications should ensure that they are not linking to static versions of any required libraries, as library versions will change. If you’re using a custom PHP.ini file or if you’re unsure, read the instructions on updating it here.

 

Here is the list of web servers to be upgraded. You can check if you’re one of these web servers by clicking the ‘Account Status’ link within the DreamHost panel and looking for ‘web server’:

 

abraham-baldwin
acrobat
albemarle
amelia
amherst
appomattox
arthur-dent
augusta
backseat
bamboo
bandbox
banks
bartow
baseball
bath
benjamin-harrison
bigtop
blacktop
blueprint
bombur
bookstore
bryan
buchanan
buckingham
bulloch
centurion
challenger
charles-pickney
charlotte
chatham
cherokee
claw
coachwood
coffee
columbia
cook
coweta
crawford
crisp
crystal
dain
daniel-carroll
deacon
diamond
dire
durin
eddie
effingham
elrond
essex
evergreen
fadeaway
fan
fannin
fayette
fencing
fireplug
flivver
ford-prefect
frederick
giles
glascock
gloin
glynn
gwinnett
habersham
hall
halo
hanover
haralson
henrico
henry
highland
hotshot
instructor
james-city
jenkins
josiah-bartlett
kili
king-william
lancaster
lee
long
loudoun
lunenburg
magnolia
marion
miller
morgan
morinehtar
nighthawk
nori
northumberland
patroller
pickens
pierce-butler
pike
pivot
polk
powhatan
prince-edward
putnam
randolph
regor
rockingham
sadalmelik
saiph
samuel-chase
sandbox
saruman
schley
scott
seminole
sheliak
skat
slacrl
snapshot
snowstorm
softpack
stardust
stewart
sugi
sussex
sycamore
talbot
templer
terrell
thrain
thror
tift
toombs
towns
tracker
traffic
trail
treutlen
tricia-mcmillan
tureis
walton
ware
warehouse
wheels
wilcox
wilkes
william-floyd
worth
wythe
zaphod-beeblebrox

 

If you have any concerns leading up to the upgrade, please don’t hesitate to contact our support team. They’ll be more than happy to answer any questions you might have!
While the upgrade is ongoing, we’ll keep this post up-to-date with new information as it becomes available. If you’d like to know more about the benefits resulting from our switch to Ubuntu, please read this post.

Update October 13th, 2014, 12:19AM PDT: The maintenance has now completed. If you find that you’re experiencing any issues with your website(s) please contact support via the link above.

Previous updates available after the jump →


8:06 pm

Upgrading 150 shared web hosting servers to Ubuntu 12.04 Precise – October 11th, 8PM – 1AM PDT (RESOLVED)

Posted (October 7th, 2014 at 8:06 pm PST) by Brandon

We’re continuing our roll-out of Ubuntu 12.04 Precise to an additional 150 web servers this Saturday, October 11th. As we’d like to get all of our customers over to this new OS, we will be upgrading 2 batches per week. While the total estimated maintenance is 5 hours, we expect actual downtime due to the upgrade to be around 45 minutes. A large part of the maintenance window will be spent testing all of the servers post-upgrade to ensure everything is in order.

As the Ubuntu image will not support PHP5.2, anyone who still has this version of PHP selected in the panel will be migrated to a Debian server before these web servers are upgraded, and will still have until the November deadline to upgrade to a more recent version of PHP.

While this upgrade process is expected to go smoothly, there are some things you can do to reduce the chances of additional service interruption. Customers running  Ruby or Python applications should ensure that they are not linking to static versions of any required libraries, as library versions will change. If you’re using a custom PHP.ini file or if you’re unsure, read the instructions on updating it here.

 

Here is the list of web servers to be upgraded. You can check if you’re one of these web servers by clicking the ‘Account Status’ link within the DreamHost panel and looking for ‘web server’:

 

accomack
alleghany
appling
arlington
atkinson
barrow
beehive
bellhop
ben-hill
berrien
bibb
birdseye
bland
bleckley
boardwalk
botetourt
brimstone
broadside
byrch
cableboy
calico
camden
cameo
campbell
candlestick
carbine
carnation
caroline
carpet
cartwheel
chandelier
chariot
charlton
chattooga
chessman
christopher
clay
clayton
coach
coppertone
culpeper
cumberland
dade
dasher
david-brearley
deepthought
derby
dog
dooly
dougherty
douglas
dusty
elbert
emanuel
fili
flag
flotus
fluvanna
footprint
forefinger
forsyth
frankie
fullback
fulton
gandalf
gargravarr
george-taylor
george-washington
gordon
grayson
greensville
gunning-bedford
horsehide
horsepower
houston
jacob-broom
james-mchenry
james-wilson
jeff-davis
jefferson
john-adams
john-hart
john-rutledge
king-and-queen
king-george
lock
lotus
louisa
lowndes
marine
mathews
mcduffie
mcintosh
mecklenburg
middlesex
mitchell
murray
muscogee
new-kent
newton
nicholas-gilman
northampton
oconee
oliver-wolcott
ori
peach
peninsula
philip-livingston
pinafore
pittsylvania
potus
professor
pulaski
rabun
radagast
radiant
rawhide
roadhouse
robert-morris
rosebush
samuel-adams
smyth
southampton
spalding
springtime
stafford
stagecoach
staircase
starlight
stephen-hopkins
stephens
store
sunburn
sunshine
surry
taliaferro
tattnall
taylor
thomas
thomas-fitzsimmons
thomas-nelson-jr
turner
volcano
washington
wayne
white
whitfield
wilkinson
william-paca
william-paterson
william-whipple

 

If you have any concerns leading up to the upgrade, please don’t hesitate to contact our support team. They’ll be more than happy to answer any questions you might have!
While the upgrade is ongoing, we’ll keep this post up-to-date with new information as it becomes available. If you’d like to know more about the benefits resulting from our switch to Ubuntu, please read this post.

Update October 12th, 2014: 1:24AM PDT: The Ubuntu upgrade on the web server ‘chessman’ has completed. The maintenance has now completed. If you find that you’re experiencing any issues with your website(s) please contact support via the link above. The following web servers were not upgraded tonight, and will be upgraded at a future date:

 

stagecoach
king-george
radagast
deepthought
broadside
dade
springtime
culpeper

Previous updates available after the jump →


11:31 am

Upgrading 150 shared web hosting servers to Ubuntu 12.04 Precise – October 5th, 8PM – 1AM PDT (RESOLVED)

Posted (September 30th, 2014 at 11:31 am PST) by Brandon

We’re continuing our roll-out of Ubuntu 12.04 Precise to an additional 150 web servers this Sunday, October 5th. As we’d like to get all of our customers over to this new OS, we’re planning on running these upgrades in larger batches. While the total estimated maintenance is 5 hours, we expect actual downtime due to the upgrade to be around 30 minutes. A large part of the maintenance window will be spent testing all of the servers post-upgrade to ensure everything is in order.

As the Ubuntu image will not support PHP5.2, anyone who still has this version of PHP selected in the panel will be migrated to a Debian server before these web servers are upgraded, and will still have until the November deadline to upgrade to a more recent version of PHP.

While this upgrade process is expected to go smoothly, there are some things you can do to reduce the chances of additional service interruption. Customers running  Ruby or Python applications should ensure that they are not linking to static versions of any required libraries, as library versions will change. Customers running custom PHP.ini setups should ensure they are pointing to PHP5.3 or higher as the PHP5.2 executable will not be present within Ubuntu.

 

Here is the list of web servers to be upgraded. You can check if you’re one of these web servers by clicking the ‘Account Status’ link within the DreamHost panel and looking for ‘web server’:

alexander-hamilton
arthur-middleton
baker
baldwin
baratheon
bedford
benjamin-franklin
benjamin-rush
benjy
bilbo
blout
brooks
buckwell
button-gwinnett
bywater
caesar-rodney
calhoun
caliber
caravan
cargo
carousel
charles-carroll
chattahoochee
chelsted
clinch
cloverleaf
cobweb
colquitt
companion
cowpuncher
craig
crossbow
crown
curbside
daniel-jenifer
dawson
dekalb
driftwood
edward-rutledge
elbridge-gerry
falcon
fenchurch
fireside
flying
fog
forward
francis-hopkinson
francis-lewis
francis-lightfoot-lee
gag-halfrunt
george-clymer
george-read
george-walton
gilmer
gouverneur-morris
hactar
halfback
handshake
harris
headlight
hedgehog
holly
huntsman
irwin
james-madison-jr
james-smith
jared-ingersoll
jasper
jeltz
john-blair
john-dickinson
john-hancock
john-langdon
john-morton
john-penn
john-witherspoon
jonathan-dayton
joseph-hewes
lamar
laurens
lewis-morris
lightfoot
lintilla
lizard
majikthise
maximillian
oglethorpe
oolon-colluphid
pacemaker
pass
pavillion
pincushion
prak
principal
rainbow
rappahannock
redfern
reliant
rhyme
ribbon
richard-bassett
richard-henry-lee
richard-stockton
ridgeline
ringside
robert-treat-paine
rockdale
roger-sherman
roosta
rufus-king
samuel-huntington
sandstone
saturn
scorecard
searchlight
shared-with-ssds
sheepskin
shotgun
slartibartfast
snowbank
strawberry
sundance
supervisor
thomas-jefferson
thomas-lynch-jr
thomas-mckean
thomas-mifflin
thomas-stone
tiller
timberwolf
tower
tranquility
troup
upson
vroomfondel
webster
wheeler
william-blount
william-ellery
william-few
william-hooper
william-livingston
william-samuel
william-williams
windstone
wowbagger
yooden-vranx
zarniwoop
zarquon
zem

 

If you have any concerns leading up to the upgrade, please don’t hesitate to contact our support team. They’ll be more than happy to answer any questions you might have!
While the upgrade is ongoing, we’ll keep this post up-to-date with new information as it becomes available. If you’d like to know more about the benefits resulting from our switch to Ubuntu, please read this post.

Update October 6th, 2014: 1:23AM PDT: All web servers are back online other than the ‘vroomfondel’ web server which is being restored to Debian due to latent hardware issues.  Baratheon, vroomfondel and zem will not be upgraded tonight.

Previous updates available after the jump →


11:26 am

Upgrading 150 shared web hosting servers to Ubuntu 12.04 Precise – October 4th, 8PM – 1AM PDT (RESOLVED)

Posted (September 30th, 2014 at 11:26 am PST) by Brandon

We’re continuing our roll-out of Ubuntu 12.04 Precise to an additional 150 web servers this Saturday, October 4th. As we’d like to get all of our customers over to this new OS, we’re planning on running these upgrades in larger batches. While the total estimated maintenance is 5 hours, we expect actual downtime due to the upgrade to be around 30 minutes. A large part of the maintenance window will be spent testing all of the servers post-upgrade to ensure everything is in order.

As the Ubuntu image will not support PHP5.2, anyone who still has this version of PHP selected in the panel will be migrated to a Debian server before these web servers are upgraded, and will still have until the November deadline to upgrade to a more recent version of PHP.

While this upgrade process is expected to go smoothly, there are some things you can do to reduce the chances of additional service interruption. Customers running  Ruby or Python applications should ensure that they are not linking to static versions of any required libraries, as library versions will change. Customers running custom PHP.ini setups should ensure they are pointing to PHP5.3 or higher as the PHP5.2 executable will not be present within Ubuntu.

 

Here is the list of web servers to be upgraded. You can check if you’re one of these web servers by clicking the ‘Account Status’ link within the DreamHost panel and looking for ‘web server’:

 

agapanthus
agba
ajuga
anemone
araucaria
ash
aster
astilbe
ayan
balsa
baptisia
barrenwort
basswood
bergenia
bloodroot
bloodwood
bluestar
boltonia
boxelder
boxwood
brazilwood
bubinga
buckeye
butternut
campion
candytuft
carapa
catmint
cattail
centaurea
chives
clematis
cocobolo
coleus
coneflower
coralbells
corkwood
cottonwood
cucumbertree
cypress
delphinium
dogwood
elm
eucalyptus
evolvulus
fennel
fir
fleabane
foamflower
fountaingrass
foxglove
fuchsia
gaura
gazania
gentian
geranium
goatsbeard
goldenrod
goldenstar
greenheart
grenadilla
gum
gunnera
hackberry
helenium
hellebore
hemlock
hibiscus
honesty
hosta
ipe
iroko
ironwood
jacaranda
kiwi
lantana
lavender
lemongrass
liverleaf
lupine
malva
maple
melon
merbau
mpingo
muscida
navi
nekkar
neshmet
nunki
oak
padauk
palisander
peacock
phad
pleione
procyon
propus
proxima-centauri
pulcherrima
ramin
rasalas
rastaban
redcedar
redheart
rigel
rotanev
ruchba
rukbat
sabik
sadachbia
sadalbari
sadalsuud
sadr
sal
salm
sandalwood
santamaria
sargas
sarin
sassafras
sceptrum
scheddi
schedir
seginus
sham
sheratan
situla
spica
spruce
subra
sulafat
syrma
tabit
talitha-australis
talitha-borealis
tambotie
tarazet
taygeta
teak
tegmen
terebellum
unuk
ushakaron
vindemiatrix
walnut
wasat
wazn
wezen
zuben-el-akrab

If you have any concerns leading up to the upgrade, please don’t hesitate to contact our support team. They’ll be more than happy to answer any questions you might have!
While the upgrade is ongoing, we’ll keep this post up-to-date with new information as it becomes available. If you’d like to know more about the benefits resulting from our switch to Ubuntu, please read this post.

Update October 5th, 2014, 12:11AM PDT: The upgrades have successfully completed. The following servers will be re-scheduled as soon as possible:

 

lupine, fountaingrass, oak, sassafrass, and gunnera

Update October 5th, 2014, 12:02AM PDT: The Ubuntu upgrades for tonight have completed. The following servers will be upgraded at a later date:

lupine, fountaingrass, oak, sassafrass, and gunnera

Previous updates available after the jump →


9:26 am

Upgrading 100 shared web hosting servers to Ubuntu 12.04 Precise – September 28th, 8PM – 1AM PDT (RESOLVED)

Posted (September 22nd, 2014 at 9:26 am PST) by Brandon

We’re continuing our roll-out of Ubuntu 12.04 Precise to an additional 100 web servers this week. As we’d like to get all of our customers over to this new OS, we’re planning on running these upgrades in larger batches. While the total estimated maintenance is 5 hours, we expect actual downtime due to the upgrade to be around 30 minutes. A large part of this time will be spent testing all of the servers post-upgrade to ensure everything is in order.

As the Ubuntu image will not support PHP5.2, anyone who still has this version of PHP selected in the panel will be migrated to a Debian server before these web servers are upgraded, and will still have until the November deadline to upgrade to a more recent version of PHP.

While this upgrade process is expected to go smoothly, there are some things you can do to reduce the chances of additional service interruption. Customers running  Ruby or Python applications should ensure that they are not linking to static versions of any required libraries, as library versions will change. Customers running custom PHP.ini setups should ensure they are pointing to PHP5.3 or higher as the PHP5.2 executable will not be present within Ubuntu.

 

Here is the list of web servers to be upgraded. You can check if you’re one of these web servers by clicking the ‘Account Status’ link within the DreamHost panel and looking for ‘web server’:

ailsabay
aken
alkan
amun
amunet
anhur
anuket
bach
ballechin
balvenie
bartok
berlioz
berwald
bes
brahms
brown
bunnahabhain
buri
caolila
cardhu
chopin
clementi
clynelish
copland
daftmill
dailuaine
demeij
dvorak
eir
eisler
el-dabh
elli
eostre
fitch
freya
frigg
fulla
gefjun
glenburgie
gliere
gorecki
haydn
henet
hlin
honegger
horus
hummel
inchgower
iusaaset
ivanovs
ives
kaelin
karlowicz
kashif
kebechet
khepri
koechlin
koppel
lagavulin
lange-muller
liszt
loch-lomond
lofn
mani
martinu
mccabe
mendelssohn
menezes
menhit
milhaud
nanna
north-british
nott
pakhet
panufnik
parry
part
port-charlotte
prokofiev
qebui
renenutet
rontgen
rubinstein
sallinen
schumann
shapiro
shostakovich
speyburn
strathearn
tchaikovsky
teaninich
the-speyside
tomatin
tomintoul
tormore
villa-lobos
whear
willson
wingate
witt

 

If you have any concerns leading up to the upgrade, please don’t hesitate to contact our support team. They’ll be more than happy to answer any questions you might have!
While the upgrade is ongoing, we’ll keep this post up-to-date with new information as it becomes available. If you’d like to know more about the benefits resulting from our switch to Ubuntu, please read this post.

Update September 28th, 2014, 3:11AM PDT:  Alright DreamHosters, thanks for staying up with us tonight! We’ve finished testing and are calling this all clear! If you run into any issues on your sites that you believe may be due to this upgrade, please contact our Technical Support team. They’ll be happy to help!

Previous updates available after the jump →


4:26 pm

Upgrading seventeen shared web hosting servers to Ubuntu 12.04 Precise – September 14th, 9PM – 12AM PDT (RESOLVED)

Posted (September 12th, 2014 at 4:26 pm PST) by Brandon

After a lot of hard work including preparations and thorough testing from our Systems Engineering team, we’re ready to move forward with upgrading our shared web servers from Debian 6 Squeeze to Ubuntu 12.04 Precise!

Upgrading to Ubuntu from Debian will provide a host of benefits, including the ability to maintain a stable release for a much longer time, while having security patches and backported packages available to our customers. With all that, we felt it was worth the efforts of making such a significant change.

We’ve upgraded six web hosting machines to Ubuntu so far, and things have gone swimmingly.

We’re now going to upgrade seventeen additional shared web servers:

applewood
artemisia
avens
bluebeard
bunchberry
burnet
columbine
coreopsis
deadnettle
dianthus
flax
globeflower
lovegrass
meranti
mint
pherkad
wenge

 

We estimate the maintenance window to be three hours, but expect around 30 minutes of actual downtime. You can check if you’re one of these web servers by clicking the ‘Account Status’ link within the DreamHost panel and looking for ‘web server’.

If you have any concerns leading up to the upgrade, please don’t hesitate to contact our support team. They’ll be more than happy to answer any questions you might have!
While the upgrade is ongoing, we’ll keep this post up-to-date with new information as it becomes available. If you’d like to know more about the benefits resulting from our switch to Ubuntu, please read this post.

Update September 14th, 2014, 11:45 PM PDT: We’ve verified that everything is running properly, and we’re now calling this complete!

Previous updates available after the jump →


10:47 am

WordPress 4.0 Major Release Now Available

Posted (September 4th, 2014 at 10:47 am PST) by Mika E.

Baboom!

Can you believe we’re on 4.0 already?

For all the wicked details, check out the official WordPress release notes for 4.0 “Benny”. I’m in love with the new media library gallery! Getting to see all my images like that makes it easier for me to see if I’ve already uploaded that cool picture of my cat as a beard before I upload it again for a new post. If you’re one of those people who adored the old way, though, and had extra columns of joy, don’t worry. You can switch right over and change your view. Also the way the post editor sizes itself to match my screen has made it easier to write epic long posts about how my cat hates being a beard, and that’s how I got this scar on my face.

Over the next few days, this update will be automatically handled for all customers who have automatic upgrades enabled, including those on DreamPress. If you used our One-Click Installer to install WordPress, you can check your settings, and change these preferences directly, by visiting the DreamHost One-Click Panel. Once there, click “Manage Installed Applications,” select the domain, then look under the Upgrade Action column for details.

By the way, WordPress uses semantic versioning, so in WP speak, 3.9 and 4.0 are the same level of absolute super-awesome. Don’t let that scary point-oh version number freak you out; we’re WordPress, not Windows.

If you run into issues after your upgrade, remember to report any problems you have with specific plugins and themes to those developers on WordPress.org (or the dev’s website)! The WordPress core devs work really hard to make sure everyone tests, but you know how real life gets in the way of real life? It happens.


 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
10:58 am

Upgrading five shared web hosting servers to Ubuntu 12.04 Precise – September 4th, 9PM – 12AM PDT(RESOLVED)

Posted (September 3rd, 2014 at 10:58 am PST) by Brandon

After a lot of hard work including preparations and thorough testing from our Systems Engineering team, we’re ready to move forward with upgrading our shared web servers from Debian 6 Squeeze to Ubuntu 12.04 Precise!

The main reason for upgrading from Debian to Ubuntu is that it will provide a host of benefits, including the ability to maintain a stable release for a much longer time, while having security patches and backported packages available to our customers. With all that, we felt it was worth the efforts of making such a significant change.

We’ve already completed the upgrades on some backend machines including  VPS host machines, mail servers and the shared web server ‘daugherty’.  As the upgrade for ‘daugherty’ went as planned, we’re now going to upgrade five additional shared web servers:

 

whale
talisker
laphroaig
nephthys
scriabin

We estimate the maintenance window to be three hours, but expect around 30 minutes of actual downtime. You can check if you’re one of these web servers by clicking the ‘Account Status’ link within the DreamHost panel and looking for ‘web server’.

If you have any concerns leading up to the upgrade, please don’t hesitate to contact our support team. They’ll be more than happy to answer any questions you might have!
While the upgrade is ongoing, we’ll keep this post up-to-date with new information as it becomes available. If you’d like to know more about the benefits resulting from our switch to Ubuntu, please read this post.

Update September 4th, 2014, 10:35 PM PDT: We’ve verified that everything is running properly, and we’re now calling this complete!

Previous updates available after the jump →


12:17 am

DreamObjects Connectivity Issues (RESOLVED)

Posted (August 29th, 2014 at 12:17 am PST) by Steven

Our cloud engineering team is investigating connectivity issues being reported by the DreamObjects service. We do not have an estimate as to how long it will take to restore services. All data stored in DreamObjects is safe and we will continue to update this post as we gather more information.

Please note, this does not affect any other DreamHost services, this is specific to DreamObjects.

If you have any questions or concerns, please don’t hesitate to contact support for assistance.

Update August 29th, 2014, 02:15 AM PDT: All tests have completed successfully! Please contact support if you still see any issues with the DreamObject service.

Update August 29th, 2014, 02:15 AM PDT: DreamObjects connectivity has been restored! Tests are still running, though, so we can be sure there are no remaining issues.

Update August 29th, 2014, 02:06 AM PDT: Work continues to resolve the problem.

Update August 29th, 2014, 01:06 AM PDT: There isn’t anything new to report at this time. Work continues to get this fixed as quickly as possible. We thank you for your patience!

Update August 29th, 2014, 12:45 AM PDT: The trouble appears to be a cluster reporting it is full when it is not. We are working to resolve this but still no ETA as of this moment.


10:34 am

DreamObjects Intermittent Delays ( RESOLVED )

Posted (August 26th, 2014 at 10:34 am PST) by Justin@DH

DreamObjects is experiencing higher than normal traffic and experiencing intermittent delays in processing requests. Our Cloud Engineering team is continuing to monitor the situation and make adjustments as needed. All data stored in DreamObjects is safe at this time. We will continue to update this post as we gather more info.

Please note, this does not affect any other DreamHost services, this is specific to DreamObjects.

If you have any questions or concerns, please don’t hesitate to contact our support team for assistance.

Update August 26th, 2014, 3:57 PM PDT: DreamObjects service has been stable for the last 4 1/2 hours. Our Cloud Engineering team will continue their routine monitoring for any further traffic spikes.

Previous updates available after the jump →


4:00 pm

Upgrading shared web hosting server ‘daugherty’ to Ubuntu 12.04 Precise – August 27th, 9PM – 12AM PDT(RESOLVED)

Posted (August 25th, 2014 at 4:00 pm PST) by Brandon

After a lot of hard work including preparations and thorough testing from our Systems Engineering team, we’re ready to move forward with upgrading our shared web servers from Debian 6 Squeeze to Ubuntu 12.04 Precise!

The main reason for upgrading from Debian to Ubuntu is that it will provide a host of benefits, including the ability to maintain a stable release for a much longer time, while having security patches and backported packages available to our customers. With all that, we felt it was worth the efforts of making such a significant change.

We’ve already completed the upgrades on some backend machines including  VPS host machines and mail servers, but we’re not taking this upgrade for granted. In order to ensure things go as smoothly as possible, we’ll be starting this upgrade on a single server, ‘daugherty’.

We estimate the maintenance window to be three hours, but expect around 30 minutes of actual downtime. You can check if you’re on the ‘daugherty’ web server by clicking the ‘Account Status’ link within the DreamHost panel and looking for ‘web server’.

If you have any concerns leading up to the upgrade, please don’t hesitate to contact our support team. They’ll be more than happy to answer any questions you might have!
While the upgrade is ongoing, we’ll keep this post up-to-date with new information as it becomes available. If you’d like to know more about the benefits resulting from our switch to Ubuntu, please read this post.

Update August 27th, 2014, 10:05 PDT:  The upgrade of ‘daugherty’ to Ubuntu has completed without issue. Sites are now back online. If you’re seeing any issues with your sites, please don’t hesitate to contact our support team.

Previous updates available after the jump →


5:40 pm

Live Chat Unavailable Tuesday, August 26th, 12am to Wednesday, August 27th, 12am PDT

Posted (August 21st, 2014 at 5:40 pm PST) by Brian@dreamhost

Our Live Chat support system will be unavailable this coming Tuesday, August 26th, from 12am to 12am (the following morning, 24hours). We will be involving our support staff in our quarterly team meeting. We’ll be taking a company-wide look at the last few months and discuss our performance and improvements moving forward. Our aim is to offer the best service possible and we feel these changes will only help to improve our Live Chat team’s performance

Email support operations will still be available, should you need to get in touch with us. We will resume normal Live Chat operations at 12am PDT the next morning, August 27th. Thank you for your patience and understanding.


1:35 pm

Scheduled Network Maintenance in US-West (Los Angeles) Data Center – August 21st 10pm PDT (4 hrs) (RESOLVED)

Posted (August 20th, 2014 at 1:35 pm PST) by Brian@dreamhost

We will be conducting network maintenance in our US-West (Los Angeles) Data Center starting at 10pm PDT. This is to perform routine maintenance on some of our network hardware. We estimate the maintenance window to be 4hrs, during which we will move connectivity from one piece of hardware to a temporary piece, then perform the maintenance, then move back. Each of these moves may cause up to 30 second of service pause, and we will be closely monitoring the process (which has already been tested) to ensure that all services come back up after the move as expected.

Please note, this maintenance will only be conducted on a section of the data center where the MySQL (shared or VPS) machines are located. In order to check if you will be affected, please check DreamHost panel for the location of your MySQL server. If you host dynamic web sites with us and your mysql server is at the Los Angeles location, those may also be affected indirectly, but there will be no impact on static sites or email service during this time.

We appreciate your patience and understanding while we continue improving our networking system. We will open the comments as soon as we start the maintenance. Until then, if you have any questions or concerns, please let us know by contacting tech support.

Update August 21st, 2014, 10:41 PM PDT: All mysql machines have been moved back to their starting router, and are now functioning correctly. We are concluding this maintenance as of now!

Previous updates available after the jump →


10:23 am

DreamObjects Intermittent Connectivity Issues (RESOLVED)

Posted (August 13th, 2014 at 10:23 am PST) by Justin@DH

Our cloud engineering team is investigating intermittent connectivity issues being reported by the DreamObjects service. There is not yet an estimate as to how long it will take to restore services. All data stored in DreamObjects is safe at this time. We will continue to update this post as we gather more info.

Please note, this does not affect any other DreamHost services, this is specific to DreamObjects.

If you have any questions or concerns, please don’t hesitate to contact support for assistance.


1:46 pm

WordPress 3.9.2 – Security Release

Posted (August 6th, 2014 at 1:46 pm PST) by Mika E.

WordPress 3.9.2 is now available as a security release for all previous versions of WordPress, and is being applied now to all sites who allow for Background Updates in WordPress, as well as any who use our Auto Update via Panel.

You can read the release notes for 3.9.2, and this release is coordinated with a Drupal release, as both projects share a potential vulnerability (due to an issue in PHP) that could cause a denial of service attack.

As a reminder, WordPress itself performs background updates of all minor releases, since WordPress 3.7 was released in 2013. That means even if you have disabled auto updates via panel (which you’d never do, because updating is the best thing since sliced bread), WordPress will still upgrade your 3.7, 3.8, and 3.9 installs to the latest, most secure, version.

If for whatever reason you went and disabled all possible update methods to your site, please log in to your WordPress site now, head to Dashboard → Updates and simply click “Update Now.” It’s just that easy.

Speaking of updates …. WordPress 4.0 is due out by the end of August! Now’s the time to have a look at your sites and make sure all your plugins and themes are up-to-date in preparation for some cool new features. You may even want to upgrade your PHP settings in panel to PHP 5.4, just to be extra cool.


4:28 pm

Scheduled Network Maintenance in US-West (Los Angeles) Data Center – July 24th 10pm PDT (4 hrs) (RESOLVED)

Posted (July 21st, 2014 at 4:28 pm PST) by andrea

Following in the path of the successful network maintenance that increased network redundancy and capacity for the MySQL servers in our US-West (Los Angeles) data center, our next step is to repeat the process for our shared web servers in that same location.

The maintenance will happen on July 24th, starting at 10pm PDT. We estimate the maintenance window to be 4hrs, during which we will be moving groups of servers from the old network uplink switch to the new ones. Each of these moves may cause up to 30 second of service pause, and we will be very closely monitoring the process (which has already been tested) to ensure that all services come back up after the move as expected.

Please note, this maintenance will only be conducted on a section of the data center where the shared web servers are located. In order to see if you will be affected, please see check the DreamHost panel for the location of your shared web server. There will be no impact on MySQL or email services during this time.

We appreciate your patience and understanding while we continue improving our networking system. We will open the comments as soon as we start the maintenance. Until then, if you have any questions or concerns, please let us know by contacting tech support.

Update July 24th, 2014, 11:44 PM PDT: We are happy to report that the maintenance completed successfully, way ahead of schedule and without noticeable service impact. We are closing the comments, but if you have any questions or need any assistance, please let us know by contacting tech support.

Previous updates available after the jump →


4:22 am

Live Chat Service Interruption (RESOLVED)

Posted (July 20th, 2014 at 4:22 am PST) by Steven

It appears the provider for our Live Chat service is experiencing trouble at this time. The service has been up and down over the last couple of hours. Please be aware that if you do connect with a tech. support representative it is possible you may be disconnected (it’s happening on our end, too). Thank you for your patience and understanding.

Update July 20th, 2014, 5:10 AM PDT: We have confirmed with LivePerson that the issue was indeed on their end and has been resolved. We do not anticipate any further trouble with the chat service but definitely let us know if you encounter any errors!


 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
12:31 am

Connectivity Issues for subset of customers in US-West Data Center (RESOLVED)

Posted (July 19th, 2014 at 12:31 am PST) by Steven

We are currently experiencing some power related issues with one of our racks in our Los Angeles data center. We apologize for the trouble and are working as quickly as possible to restore service. The servers impacted by this are:

hindemith, holmboe, honegger, hovhaness, hummel, dindy, ivanovs, ives, kaelin, karlowicz, kashif, koechlin, koppel, lalo
liszt, lange-muller, martinu, mahler, melartin, mccabe, mendelssohn, menezes, messiaen, mignone, milhaud, mozart
nielsen, nystroem, panufnik, parry, part, penderecki, persichetti, prokofiev, delpuerto

You can lookup which server you are hosted on in the admin panel by clicking the Account Status link at the top-right.

Watch for additional updates over the next half to one hour. If you have any questions or need anything else please Contact Support. Thank you!

Update July 19th, 2014, 1:40 AM PDT: This is resolved now and all servers are up. Once again, we thank you for your patience. If you still see any problems with your service please contact support through the link above. Thanks!

Update July 19th, 2014, 1:11 AM PDT: We are still working on the problem. Unfortunately, we are unable to provide an ETA for resolution at this time. Thank you for your patience and we will provide another update as soon as possible.


11:59 am

Limited service availability at our US-West (Los Angeles) Data Center (RESOLVED)

Posted (July 16th, 2014 at 11:59 am PST) by andrea

We are currently experiencing connection issues at sections of our US-West (Los Angeles) data center. Our operations teams are already looking into it, and we’ll update this status post as soon as we have more information. Very sorry about the inconvenience this is causing.

Update July 16th, 2014, 2:32 PM PDT: We are happy to report that all affected services have been fully restored, including mail and shared web servers.
While working on this, we also took steps towards avoiding similar situations in the future: we rerouted the shared web services from the affected panel to a different power source and moved some redundant circuits off as well to further alleviate load on said panel. Again, please accept our apologies for the inconvenience this has caused. If you have any questions or concerns, please contact tech support, and we’ll be happy to help!

Previous updates available after the jump →


1:49 pm

Scheduled Network Maintenance in US-West (Los Angeles) Data Center – July 10th 10pm PDT (4 hrs) RESOLVED

Posted (July 8th, 2014 at 1:49 pm PST) by andrea

We will be conducting network maintenance in our US-West (Los Angeles) Data Center starting at 10pm PDT. This is to upgrade some of our current equipment to increase network redundancy and capacity. We estimate the maintenance window to be 4hrs, during which we will be moving groups of servers from the old network uplink switch to the new ones. Each of these moves may cause up to 30 second of service pause, and we will be very closely monitoring the process (which has already been tested) to ensure that all services come back up after the move as expected.

Please note, this maintenance will only be conducted on a section of the data center where the MySQL (shared or VPS) machines are located. In order to check if you will be affected, please check the DreamHost panel for the location of your MySQL server. If you host dynamic web sites with us and your mysql server is at the Los Angeles location, those may also be affected indirectly, but there will be no impact on static sites or email service during this time.

We appreciate your patience and understanding while we continue improving our networking system. We will open the comments as soon as we start the maintenance. Until then, if you have any questions or concerns, please let us know by contacting tech support.

Update July 11th, 2014, 1:09 AM PDT: We are happy to report that the maintenance, including the follow-up testing phase, has been successfully completed, with no noticeable impact on related services. Thank you again for your patience while we were working on it! We are closing the comments, but if you have any questions or concerns, please feel free to contact tech support, and we’ll be happy to help.

Previous updates available after the jump →


12:17 pm

Network infrastructure maintenance – June 19th 6PM PDT (2 hours)

Posted (June 19th, 2014 at 12:17 pm PST) by Brandon

On Thursday June 19th, starting at 6:00PM PDT we will be replacing one of the network switches in our mail infrastructure. We expect no major impact to customers accessing mail. This is a proactive maintenance, and the previous maintenance in this series was executed seamlessly. We expect this maintenance to perform the same.

Update, 6:43 PM Pacific Daylight Time: And everything is complete! All went as planned. If you have any questions regarding this, please let us know by contacting Support.

Previous updates available after the jump →


10:57 pm

Power issues in our US-West Data Center (Los Angeles) (RESOLVED)

Posted (June 13th, 2014 at 10:57 pm PST) by Steven

We have experienced two power outages lasting 20 minutes or longer on one of our racks in the Los Angeles data center. There are twelve servers that have been impacted by this:

hovhaness, kaelin, mozart, hindemith, honegger, nielsen, milhaud, nystroem, holmboe, koechlin, kashif, and karlowicz.

We are actively investigating the cause of the outages so we can ensure they do not continue to occur.


9:57 am

DreamObjects Outage (RESOLVED)

Posted (June 13th, 2014 at 9:57 am PST) by Dan J

Our cloud engineering team are currently investigating 503 errors being reported by the DreamObjects service. At the moment it appears some servers are unresponsive and that is causing the errors. There is not yet an estimate as to how long it will take to restore services. All data stored in DreamObjects is safe at this time. We will continue to update this post as we gather more info.

Please note, this does not affect any other DreamHost services, this is specific to DreamObjects.

If you have any questions or concerns, please don’t hesitate to contact support contact support for assistance.

Update June 14th, 2014, 10:20 AM PDT: Stability on the DreamObjects cluster has been restored. Requests appear to be resolving properly now that the system has had time to re-balance itself. Our test are reporting properly now. If you do have any questions or concerns, please contact support contact support.

Previous updates available after the jump →


3:59 pm

WebFTP Upgrade – June 10th 11pm PDT (1 hours) (RESOLVED)

Posted (June 6th, 2014 at 3:59 pm PST) by Brian@dreamhost

On Tuesday June 10th, starting at 11:00pm PDT we will be upgrading our WebFTP software. WebFTP (https://webftp.dreamhost.com) will be temporarily unavailable as we make this change. The duration of this maintenance is 1 hour, from 11pm PDT till 12am PDT June 11ith. If you have any questions about the upcoming maintenance, please contact our support team.

Update June 10th, 2014, 11:45 PM PDT: The upgrade has been successfully completed! There should be no issues as a result. Be sure, however, to contact us if you see any strange behavior in the WebFTP software. Thanks!

Update June 10th, 2014, 11:00 PM PDT: The WebFTP service will be offline shortly to upgrade the software. As noted, we only expect this to take about one hour. Please contact support if you have any questions or concerns.


12:30 am

DreamObjects Outage (RESOLVED)

Posted (June 6th, 2014 at 12:30 am PST) by chalise

We are currently experiencing an outage with our DreamObjects service. We will update with more information as it becomes available.

Please note, this does not affect any other DreamHost services, this is specific to DreamObjects.

If you have any questions or concerns, please don’t hesitate contact support contact support for assistance.

Update June 6th, 2014, 8:33 AM PDT:  OK DreamObjects is online. Our automated tests are passing and the cluster is in its expected state.  If you do find anything that isn’t functioning as expected then please don’t hesitate contact support contact support for assistance.

Previous updates available after the jump →


4:50 pm

Time for PHP Upgrades!

Posted (June 5th, 2014 at 4:50 pm PST) by mir

Starting June 24th, we will be upgrading domains on shared hosting running PHP 5.2* to our recommended version of PHP 5.4.

Fear not! If you need a little more time to upgrade, you can delay until November by going to the Dashboard of the DreamHost Panel and selecting “see what sites need upgrades” and by pressing “delay”. For full information about the upgrade, and how to delay (and even how to tell if you need to delay), please check out this upgrading PHP article in our wiki.

*PHP 5.2 has been completely unsupported by PHP.net since January 2011, so if you were planning on delaying, we strongly recommend that you begin testing right away.

As always, if you have any questions or concerns don’t hesitate to contact our support team.


11:53 pm

Network Maintenance Affecting US-East (Virginia) June 5th 9pm – June 6th 1am PST(4 hours) (RESOLVED)

Posted (June 3rd, 2014 at 11:53 pm PST) by Brian@dreamhost

On Thursday, June 5th, starting at 9pm PST we will be performing network maintenance on one of our data centers:

US-East (Virginia)

This maintenance will last 4 hours, and improve communication between our data centers and the internet at large.

The changes will require propagation across the internet via the BGP routing protocol. As a result, your service may appear to be down during the maintenance for a short period that should not exceed 1 minute. Service interruption will largely depend on your ISP and how quickly the changes register within their network. To help identify any problems, we will have our staff monitoring the changes from multiple locations and ISPs.

We will provide updates as we move through the maintenance.

To see which data center your site is hosted in, please look at Status->Data Centers section in the panel.

Comments will be opened when the maintenance starts so that we can assist you with any issues you may be noticing.

If you have any questions about the upcoming maintenance, please contact our support team.

Update June 5th, 2014, 10:09 PM PDT: And just like that, we’re done! We thank you for your patience! If you’re seeing any issues, please contact our support team right away!

Previous updates available after the jump →


 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
8:40 am

Webserver Hardware Upgrades in US-West (Los Angeles) Datacenter – Beginning May 30th

Posted (May 27th, 2014 at 8:40 am PST) by Justin@DH

On Monday, May 12th and continuing for two weeks, we will be performing webserver hardware upgrades on certain shared webservers and web VPS host machines in our US-West (Los Angeles) datacenter. This process is necessary to replace older hardware with newer and more efficient hardware. Customers affected by this move will be sent a notification email the week prior and an email when their move is complete with additional details on what changes may be necessary afterwards.

The following shared webservers will be affected:

banshee biggs bulger burke castiglia colossus coonan cummiskey dagger dalitz deadpool dio fay fitzpatrick floyd galactus greengoblin havok hawkeye hoffa joker jubilee kakuji karpis mclean metropolis mothman nightwing pistone provenzano spiderman spillane storm walsh watanabe wolverine zwerbach

As well as these web VPS host machines:

homie-vserver02 homie-vserver06 homie-vserver47 homie-vserver48 homie-vserver55 homie-vserver56

You can verify which shared webserver or VPS host machine you are hosted on in the control panel data centers page. In case of VPS host machine, the details column lists the host machine for your VPS(s). If you have any questions please contact support.


10:17 am

DreamObjects Maintenance – May 28th 10AM PDT – 12PM PDT (2 hrs) (EXTENDED) (RESOLVED)

Posted (May 22nd, 2014 at 10:17 am PST) by Justin@DH

As part of our continued efforts to improve our cloud services, on Wednesday May 28th from 10am-12pm PDT, our Systems Engineering team will be performing maintenance on our DreamObjects service. During this maintenance window, you may intermittently experience slower upload and access speeds and some short periods of connectivity loss.

During this two hour maintenance window, we will be upgrading Ceph, the underlying storage technology powering DreamObjects. The upgrade includes improvements for large object support when using the OpenStack Swift API.

While we anticipate increased latency when storing or retrieving objects during this maintenance, please rest assured that your data is safe and will not be affected. We will closely monitor the maintenance and will be posting periodic updates.

If you have any questions or concerns, please don’t hesitate contact support contact support for assistance.

Update May 28th, 2014, 2:37 PM PDT: During testing of the DreamObjects upgrade, we found a bug that forced us to abandon the upgrade and continue at a later date. Once the issue is resolved, we will schedule another upgrade window. We apologize for this, and thank you for your patience!”
Previous updates available after the jump →


1:13 pm

Webserver Hardware Upgrades in US-West (Los Angeles) Datacenter – Beginning May 12th

Posted (May 9th, 2014 at 1:13 pm PST) by Justin@DH

On Monday, May 12th and continuing for two weeks, we will be performing webserver hardware upgrades on certain shared webservers and web VPS host machines in our US-West (Los Angeles) datacenter. This process is necessary to replace older hardware with newer and more efficient hardware. Customers affected by this move will be sent a notification email the week prior and an email when their move is complete with additional details on what changes may be necessary afterwards.

The following shared webservers will be affected:

aquaman barker batman bonanno brasco bruno buchalter bugsy cable capone castellano catwoman cloak cohen cyclops dhaccountbackup eastman escobar gambino gambit genovese giancana gotti gravano greene hulk iceman kingpin lansky legs lepke magneto mangano masseria mcgurn moran nitti obelix penguin phantom riddler robin rothstein scalise schultz siegel spilotro sugeknight terranova thing trafficante venom

As well as these web VPS host machines:

homie-vserver01 homie-vserver04 homie-vserver05 homie-vserver07 homie-vserver08 homie-vserver09 homie-vserver10 homie-vserver11 homie-vserver12 homie-vserver13 homie-vserver139 homie-vserver14 homie-vserver15 homie-vserver152 homie-vserver16 homie-vserver17 homie-vserver177 homie-vserver18 homie-vserver19 homie-vserver197 homie-vserver20 homie-vserver21 homie-vserver22 homie-vserver23 homie-vserver230 homie-vserver24 homie-vserver25 homie-vserver26 homie-vserver27 homie-vserver28 homie-vserver29 homie-vserver30 homie-vserver31 homie-vserver32 homie-vserver33 homie-vserver34 homie-vserver35 homie-vserver37 homie-vserver396 homie-vserver42 homie-vserver44 homie-vserver45 homie-vserver46 homie-vserver53

You can verify which shared webserver or VPS host machine you are hosted on in the control panel data centers page. In case of VPS host machine, the details column lists the host machine for your VPS(s). If you have any questions please contact support.


8:04 am

Web VPS Maintenance in US-West (Los Angeles) – May 14th 10PM PDT (2 hrs) (RESOLVED)

Posted (May 7th, 2014 at 8:04 am PST) by Justin@DH

On Tuesday, May 13th from 10PM to 12AM PDT, we will be performing maintenance on certain web VPS host machines in our US-West (Los Angeles) datacenter. To complete the process those host machines will need to be rebooted (which can take up to 20 minutes from start to finish). While this will result in brief service disruption, the reboot is necessary and will allow us to implement new features which will streamline future migration of web VPS services to different hardware without service interruption. In particular, this will allow us to replace older hardware newer and more efficient hardware (that change is in the works!). The following web VPS host machines will be affected:

homie-vserver01 homie-vserver04 homie-vserver05 homie-vserver07 homie-vserver09 homie-vserver10 homie-vserver11 homie-vserver12 homie-vserver13 homie-vserver14 homie-vserver15 homie-vserver16 homie-vserver17 homie-vserver18 homie-vserver19 homie-vserver20 homie-vserver22 homie-vserver23 homie-vserver25 homie-vserver26 homie-vserver28 homie-vserver29 homie-vserver30 homie-vserver31 homie-vserver32 homie-vserver33 homie-vserver34 homie-vserver35 homie-vserver42 homie-vserver44 homie-vserver45 homie-vserver46 homie-vserver53 homie-vserver197 homie-vserver396

You can verify which VPS host machine you are hosted on in the control panel data centers page. The details column lists the host machine for your VPS(s). If you have any questions please contact support.

We will update this post when the maintenance starts and with the progress. Thank you for your patience while we upgrade your services.

Update May 14th, 2014, 10:57 PM PDT: Tonight’s maintenance is complete!

Update May 14th, 2014, 10:30 PM PDT: We apologize for the short notice as we are going to proceed with this maintenance tonight. We are getting under way in just a few minutes!

Update May 13th, 2014, 10:39 PM PDT: We are postponing this for another day (to be determined).


4:51 pm

Discussion list scheduled maintenance May 7th 6pm PDT (4hrs) (RESOLVED)

Posted (May 6th, 2014 at 4:51 pm PST) by andrea

On Wednesday, May 7th, starting at 6pm, we will be performing maintenance on all our discussion list services. We will be upgrading the the Mailman software that’s running those lists from 2.1.14 to 2.1.17. Along with this upgrade we’re also turning on DKIM capabilities. This is a very important change for those of you who have been affected by Yahoo’s DMARC compliance change.

During this maintenance window your list administration interface will be unavailable. Messages posted to your list during this time will be queued and delivered later when the maintenance has been completed.

We apologize for the short notice, and we appreciate your understanding and patience while we perform this upgrade. We will open comments during the maintenance window and will be closely monitoring the progress. This update only affects discussion lists, and no other services (such as regular email, web hosting, or mysql hosting) are affected.

If you have any questions or concerns, please don’t hesitate to contact support, and we’ll be happy to help.

Update May 7th, 2014, 10:15 PM PDT:  We are happy to report that the discussion list maintenance has been completed successfully, and your discussion lists are now using Mailman version 2.1.17. This upgraded version of Mailman has the benefit of offering two options to make a discussion list message correctly pass a receiving mail server’s checks. For more information on those options and how to set them in the Mailman admin panel, please see DreamHost Wiki: Discussion Lists.

Thanks again for your patience while we were taking care of this. Please note, your list administration interface should be accessible again and your queued messages are in the process of getting delivered and should catch up shortly. If you have any questions or concerns regarding this, please contact support and let us know.

Previous updates available after the jump →


4:33 pm

Live Chat Unavailable Thursday, May 8th, 12am to Friday, May 9th, 12am PDT

Posted (May 6th, 2014 at 4:33 pm PST) by Brian@dreamhost

Our Live Chat support system will be unavailable this coming Thursday, May 8th, from 12am to 12am (the following morning, 24hours). We will be involving our support staff in team meeting. We’ll be taking a company-wide look at the last few months and discuss our performance and improvements moving forward. Our aim is to offer the best service possible and we feel these changes will only help to improve our Live Chat team’s performance

Email support operations will still be available, should you need to get in touch with us. We will resume normal Live Chat operations at 12am PDT the next morning, May 9th. Thank you for your patience and understanding.


7:02 pm

DreamObjects connectivity issues (RESOLVED)

Posted (May 4th, 2014 at 7:02 pm PST) by andrea

We are currently experiencing connectivity issues to our DreamObjects service. This includes inability to upload, download, or access CDN (DreamSpeed). Our teams are on the case, and we’re investigating the cause. We’ll update this post as soon as we have news. Please accept our apologies for the inconvenience.

This outage only affects our DreamObjects service. No other services, including hosting, email and mysql are affected.

Update Sunday May 4th 7:40pm PDT: We are happy to report that have found a resolution and all related services are functioning as expected. Please accept our apologies for any inconvenience this has caused. As always, if you have any questions or concerns, please contact support with the details, so we can assist you.

Previous updates available after the jump →


5:55 pm

DreamHost panel connectivity issues (RESOLVED)

Posted (May 4th, 2014 at 5:55 pm PST) by Seohee

We are currently experiencing connectivity issues with the DreamHost panel. We are looking into the cause now and working towards restoring functionality.

We appreciate your patience and are sorry for the inconvenience. We will provide you with an update shortly.

Update Sunday May 4th 6:40pm PDT: The issues plaguing the DreamHost panel have been resolved and all is operating again as expected. We are resolving this status post yet will continue working to find the root cause of today’s outage. As always, if you have any questions or concerns, please don’t hesitate to contact our technical support team, and we’ll be happy to help.

Previous updates available after the jump →


8:34 pm

DreamObjects connectivity issues (RESOLVED)

Posted (May 3rd, 2014 at 8:34 pm PST) by andrea

We are currently experiencing connectivity issues to our DreamObjects service. This includes inability to upload, download, or access CDN (DreamSpeed). Our teams are on the case, and we’re investigating the cause. We’ll update this post as soon as we have news. Please accept our apologies for the inconvenience.

This outage only affects our DreamObjects service. No other services, including hosting, email and mysql are affected.

Update Saturday May 3rd 11:15pm PDT: We are happy to report that have found a resolution and all related services are functioning as expected. Please accept our apologies for any inconvenience this has caused. As always, if you have any questions or concerns, please contact support with the details, so we can assist you.

Previous updates available after the jump →


3:12 pm

DreamObjects Maintenance – Software May 5th 10am PDT (8 hrs) (RESOLVED)

Posted (May 2nd, 2014 at 3:12 pm PST) by andrea

On Monday, May 5th between 10am and 6pm PDT, our Systems Engineering team will continue the maintenance that started on May 1st to upgrade Ceph, the underlying storage technology powering DreamObjects. This upgrade includes general performance improvements as well as additional features for managing the cluster.

During this maintenance window, you may intermittently experience slower upload and access speeds and some short periods of connectivity loss. However, please rest assured that your data is safe and will not be affected. We will closely monitor the maintenance and will be posting periodic updates.

If you have any questions or concerns, please don’t hesitate to contact support for assistance.

Update Monday May 6th 10:58am PDT: We are happy to report that the above maintenance is has successfully completed yesterday. We will be posting here again when the next maintenance is coming.

Previous updates available after the jump →


 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
3:36 pm

DreamObjects Maintenance – Software May 1st 6PM PDT (10+ hrs)(POSTPONED)

Posted (April 25th, 2014 at 3:36 pm PST) by andrea

As part of our continued efforts to improve our cloud services, on Thursday May 1st from 6pm-12am PDT, our Systems Engineering team will be performing maintenance on our DreamObjects service. During this maintenance window, you may intermittently experience slower upload and access speeds and some short periods of connectivity loss.

During this six hour maintenance window, we will be upgrading Ceph, the underlying storage technology powering DreamObjects. The upgrade includes general performance improvements as well as additional features for managing the cluster.

While we anticipate increased latency when storing or retrieving objects during this maintenance, please rest assured that your data is safe and will not be affected. We will closely monitor the maintenance and will be posting periodic updates.

If you have any questions or concerns, please don’t hesitate to contact support for assistance.

Update Friday May 2nd 3pm PDT: As noted in an earlier update, we will have to schedule another maintenance window to finish what we couldn’t complete last night. Sorry about any inconvenience this may have caused. Please see the new DreamObjects maintenance post for more details.

Previous updates available after the jump →


3:34 pm

DreamObjects Maintenance – Hardware April 30th 6PM PDT (6 hrs) (RESOLVED)

Posted (April 25th, 2014 at 3:34 pm PST) by andrea

As part of our continued efforts to improve our cloud services, on Wednesday April 30th from 6pm-12am PDT, our Systems Engineering team will be performing maintenance on our DreamObjects service. During this maintenance window, you may intermittently experience slower upload and access speeds and some short periods of connectivity loss.

During this six hour maintenance window, we will be upgrading hardware on the machines that oversee the storage nodes.

While we anticipate increased latency when storing or retrieving objects during this maintenance, please rest assured that your data is safe and will not be affected. We will closely monitor the maintenance and will be posting periodic updates.

If you have any questions or concerns, please don’t hesitate to contact support for assistance.

Update Thursday May 1st 11:05am PDT: This is just a quick follow up to let you know that last night’s maintenance was successful and well within the announce maintenance window. Please note, there is another DreamObjects maintenance tonight. You can find the details under the other DreamObjects post on this status page. As always, if you have any questions or concerns, please don’t hesitate to contact support for assistance.


9:39 pm

DreamObjects connection troubles (RESOLVED)

Posted (April 24th, 2014 at 9:39 pm PST) by andrea

We are currently experiencing connectivity issues to our DreamObjects service. This includes inability to upload, download, or access CDN (DreamSpeed). Our teams are on the case and we’re investigating the cause. We’ll update this post as soon as we have news. Please accept our apologies for the inconvenience.

This outage only affects our DreamObjects service. No other services, including hosting, email and mysql are affected.

Update Thursday April 25th 12:47am PDT: This issue should now be resolved. We are able to verify that all instances are now working. If you are still seeing problems, please contact support for assistance. We thank you for your patience during this issue!

Update Thursday April 25th 12:20am PDT: We are now making some progress bringing DreamObjects back online. Some Objects should appear up, however, they may be slow, or showing some errors. We are continuing to investigate these issues and appreciate your patience.

Previous updates available after the jump →


8:22 am

Web VPS Maintenance in US-West (Los Angeles) – April 29th 10PM PDT (2 hrs) (RESOLVED)

Posted (April 22nd, 2014 at 8:22 am PST) by Justin@DH

On Tuesday, April 29th from 10PM to 12AM PDT, we will be performing maintenance on certain web VPS host machines in our US-West (Los Angeles) datacenter. To complete the process those host machines will need to be rebooted (which can take up to 20 minutes from start to finish). While this will result in brief service disruption, the reboot is necessary and will allow us to implement new features which will streamline future migration of web VPS services to different hardware without service interruption. In particular, this will allow us to replace older hardware newer and more efficient hardware (that change is in the works!). The following web VPS host machines will be affected:

homie-vserver73
homie-vserver86
homie-vserver87
homie-vserver88
homie-vserver91
homie-vserver92
homie-vserver93
homie-vserver94
homie-vserver95
homie-vserver97
homie-vserver100
homie-vserver233

You can verify which VPS host machine you are hosted on in the control panel data centers page. The details column lists the host machine for your VPS(s). If you have any questions please contact support.

We will update this post when the maintenance starts and with the progress. Thank you for your patience while we upgrade your services.

Update Thursday April 29th 10:28pm PDT: And just as the party starts it gets shut down! That’s how you know it’s a real good party… Alas, this maintenance has been completed and no VPS hosts were harmed in the process.

Update Thursday April 29th 10:07pm PDT: We have begun! Downtime should be short and limited to the reboots for each VPS host machine. Stay tuned for updates!

Update Thursday April 29th 10:05pm PDT: We are about to get under way!


5:24 am

Webserver Hardware Upgrades in US-West (Los Angeles) Datacenter – Beginning April 28th

Posted (April 22nd, 2014 at 5:24 am PST) by Justin@DH

On Monday, April 28th and continuing for two weeks, we will be performing webserver hardware upgrades on certain shared webservers and web VPS host machines in our US-West (Los Angeles) datacenter. This process is necessary to replace older hardware with newer and more efficient hardware. Customers affected by this move will be sent a notification email the week prior and an email when their move is complete with additional details on what changes may be necessary afterwards.

The following shared webservers will be affected:

alouettes argonauts baku bears beijing bills bluejays brasilia bridgetown browns brussels bujumbura canberra cannons cardinals cavs chivas crew crusaders d-fenders eagles earthquakes eskimos fever fire force grizzlies herediano hurricanes indians islanders jazz jets kabul lightning lizards luanda minsk nassau nuggets oilers olympiacos ottawa ouagadougou outlaws penguins pigwidgeon pirates pistons pride rapids rockets santiago shock sofia stjohns thimphu thrashers tigercats vienna vikings waratahs yankees yaounde

As well as these web VPS host machines:

homie-vserver73 homie-vserver86 homie-vserver87 homie-vserver88 homie-vserver91 homie-vserver92 homie-vserver93 homie-vserver94 homie-vserver95 homie-vserver96 homie-vserver97 homie-vserver98 homie-vserver99 homie-vserver100 homie-vserver233

You can verify which shared webserver or VPS host machine you are hosted on in the control panel data centers page. In case of VPS host machine, the details column lists the host machine for your VPS(s). If you have any questions please contact support.


2:44 pm

DreamHost Web Panel Changes April 22nd, 2 PM PDT

Posted (April 18th, 2014 at 2:44 pm PST) by mir

We are excited to share some impending changes to your DreamHost Panel (https://panel.dreamhost.com). Next Tuesday we’ll be making a few minor organizational changes that mark the first preparations for some improvements to how you manage your DreamHost services. Based on suggestions and usage data we’re re-ordering and consolidating the navigation, moving a few items and removing some cruft. Please head here for full details (this document would be a good one to bookmark to look for any future alterations as well):

http://wiki.dreamhost.com/Panel_UI_Changelog

As always, if you have any questions or concerns, please don’t hesitate to contact support for assistance.

Thank you for choosing DreamHost!


11:52 pm

WebFTP Upgrade – April 18th 10pm PDT (4 hours) (RESOLVED)

Posted (April 17th, 2014 at 11:52 pm PST) by Oscar

On Friday April 18th, starting at 10:00pm PDT we will be replacing our WebFTP software. AjaXplorer is no longer being developed and has been replaced by Pydio! During this maintenance, we will be removing our AjaXplorer WebFTP client and installing Pydio. WebFTP (https://webftp.dreamhost.com) will be temporarily unavailable as we make this change. The duration of this maintenance is 4 hours, from 10pm PDT till 2am PDT April 19th. All WebFTP references will remain on the DreamHost panel and will link to the new software afterwards. If you have any questions or concerns, please don’t hesitate to contact support for assistance.

UPDATE: April 19, 2014, 12:40 AM PDT: I apologize for the delay on this post. The good news, however, is that we are done! The upgrade is complete and no issues were found during testing. If you do experience any problems with the WebFTP interface please contact our Technical Support team. Thanks!

UPDATE: April 19, 2014, 12:00 AM PDT: Nothing new to update at the moment. We are still working to complete the upgrade for WebFTP.

UPDATE: April 18, 2014, 11:30 PM PDT: We are still working on it. Stay tuned for further updates!

UPDATE: April 18, 2014, 11:02 PM PDT: The maintenance is moving along. We have hit one snag (webftp.dreamhost.com is loading with a blank page) but do not anticipate going outside of the quoted 4 hour window. Hang in there!

UPDATE: April 18, 2014, 10:37 PM PDT: We are a go! We are beginning work on the upgrade now and will post updates frequently (every 30 mins or less). WebFTP is currently offline.

UPDATE: April 18, 2014, 10:21 PM PDT: Just a heads up that we are currently getting prepared to move forward with the upgrade!


2:40 pm

DreamObjects Maintenance April 23rd 6PM PDT – April 24th 4AM PDT (10 hrs) (RESOLVED)

Posted (April 16th, 2014 at 2:40 pm PST) by Justin@DH

As part of our continued efforts to improve our cloud services, on Wednesday April 23rd from 6pm-4am PDT, our Systems Engineering team will be performing maintenance on our DreamObjects service. During this maintenance window, you may intermittently experience slower upload and access speeds and some short periods of connectivity loss. We will be upgrading Ceph, the underlying storage technology powering DreamObjects. The upgrade includes general performance improvements as well as additional features for managing the cluster.

While we anticipate increased latency when storing or retrieving objects during this maintenance, please rest assured that your data is safe and will not be affected. We will closely monitor the maintenance and will be posting periodic updates.

If you have any questions or concerns, please don’t hesitate to contact support for assistance.

Update Thursday April 24th 2:20pm PDT: Just a quick update to let you know that the maintenance went down without a hitch. If you have any questions or concerns about your DreamObjects service, please don’t hesitate to contact support for assistance.


11:55 am

WordPress 3.9 Major Release Now Available

Posted (April 16th, 2014 at 11:55 am PST) by Mika E.

It’s been a rough few weeks in the WordPress world, but WordPress 3.9 has finally been released.

All the security features of 3.8.3, plus some amazing new editor features are now available on the DreamHost panel for new installs and upgrades. You can also upgrade from directly within your site’s WordPress admin panel, if you prefer.

For all the details, check out the official WordPress release notes for 3.9 “Smith”. We’re most excited about the editor changes. You can drag and drop media right into the text editor, but also you can just copy and paste from Word right into WordPress. No more remembering to click on that ‘paste from Word’ icon no one can find!

Over the next few days, this update will be automatically handled for all customers who have automatic upgrades enabled, including all those on DreamPress. If you used our One-Click Installer to install WordPress, you can check your settings, and change these preferences directly, by visiting the DreamHost One-Click Panel. Once there, click “Manage Installed Applications,” select the domain, then look under the Upgrade Action column for details.

If you run into issues after your upgrade, remember to report any problems you have with specific plugins and themes to those developers on WordPress.org! Every single plugin developer was contracted, prior to release, to remind them to test, so we’re hoping that worked. It was a very large email list.


3:55 pm

Network Improvement Series Maintenance, US-West (Irvine) Data Center Thursday April 17th 10pm PDT (RESOLVED)

Posted (April 15th, 2014 at 3:55 pm PST) by Ralph

On Thursday April 17th, starting at 10pm PDT, we will be conducting network maintenance in our US-West (Irvine) Data Center. This is the final maintenance in our network refresh series, Network Improvement Series. The expected duration of this maintenance is 4 hours. During this maintenance, we will be upgrading the software on our core router. This is a tested process that we have done before with no noticeable customer impact. We anticipate similar results this time, with the slight possibility of brief downtime as we go through each rows of machines while moving links to and from the backup switches.

Our order of operations is as follows:

1. Moving links, one by one, from the main core router to our temporary switches. ( Can cause less than a minute of service downtime for sites, mail and other services, as links associated with those services are moved.)

2. Reboot the core. ( No impact.)

3. Moving the links back to the successfully rebooted core. ( Can cause less than a minute of service downtime for sites, mail and other services, as links associated with those services are moved.)

We are excited that this maintenance will conclude our efforts to refresh our network. We have already seen improvements of our network stability as we have progressed through our Network Improvement Series of Maintenances.

Comments will be open during the maintenance so that we can assist you in real time, and verify if your issues are related to this maintenance.

You can check on the DreamHost Panel if you have services located in the US-West (Los Angeles) Data Center.
If you have any questions or concerns please contact the DreamHost Support Team.

Update Friday April 18th 1:10am PDT: We are now resolving the maintenance, along with our Network Improvement Series. Tonight’s maintenance completed without any major interruption of service, and we saw no customer impact. Thank you for your patience during this update, as well as during the Network Improvement Series, which has helped us solidify our network for the future.

Previous updates available after the jump →


 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
8:55 am

Web VPS Maintenance in US-West (Los Angeles) – April 15th 10PM PDT (2 hrs) (RESOLVED)

Posted (April 10th, 2014 at 8:55 am PST) by Justin@DH

On Tuesday, April 15th from 10PM to 12AM PDT, we will be performing maintenance on certain web VPS host machines in our US-West (Los Angeles) datacenter. To complete the process those host machines will need to be rebooted (which can take up to 20 minutes from start to finish). While this will result in brief service disruption, the reboot is necessary and will allow us to implement new features which will streamline future migration of web VPS services to different hardware without service interruption. In particular, this will allow us to replace older hardware newer and more efficient hardware (that change is in the works!). The following web VPS host machines will be affected:

homie-vserver71
homie-vserver74
homie-vserver75
homie-vserver76
homie-vserver78
homie-vserver79
homie-vserver81
homie-vserver82
homie-vserver83
homie-vserver84
homie-vserver89
homie-vserver196

You can verify which VPS host machine you are hosted on in the control panel data centers page. The details column lists the host machine for your VPS(s). If you have any questions please contact support.

We will update this post when the maintenance starts and with the progress. Thank you for your patience while we upgrade your services.

UPDATE: April 15, 2014, 23:16 PDT: All done! Everything went as planned.

UPDATE: April 15, 2014, 22:50 PDT: This is progressing as expected so far with no issues. We will post another update in 30 minutes, or sooner if there is any change.

UPDATE: April 15, 2014, 22:11 PDT: We have started the maintenance work outlined above. As noted, we do not anticipate downtime for the listed host machines to last very long.


3:22 pm

DreamHost and Heartbleed – Notes on OpenSSL Vulnerability

Posted (April 9th, 2014 at 3:22 pm PST) by Brett

This vulnerability only affects domains which use secure hosting / SSL.
– If you purchased a secure certificate from us, you’re safe! There is nothing you need to do.
– If you generated keys with us but purchased the certificate from another provider, you will have to check with them to see if you need new ones. If you do, you will have to get the new certificate through them and then we’ll be happy to provide you with new keys!
– If you’ve purchased a certificate from us and are now using it with another provider, you will need to ensure that other provider was not affected by this vulnerability. If they were, our technical support team will be happy to issue you a new certificate and keys!

As soon as we learned of the “Heartbleed” OpenSSL vulnerability, we began to patch any and all systems that it may have affected. Fortunately this was a very small subset of our systems and was mostly isolated to a small group of mail machines. As of early yesterday, all of our systems are patched. As a preventative measure, we are also re-keying the certificates on any systems with that bug. We have no reason to believe that any of those machines have been compromised, but in the interest of proactive security, we feel that changing SSL certificates is the best option.

DreamHost.com was not vulnerable, but the machines that redirected traffic to our actual site were. This was corrected quickly and those machines will also have their certificates re-keyed.

We can confidently say that our shared servers, VPS guests, and dedicated machines are NOT vulnerable to this issue because they run Debian “Lenny” and/or “Squeeze”. The most common version of OpenSSL on our network is 0.9.8o-4squeeze14, and the “HeartBleed” vulnerability in OpenSSL’s heartbeat module exists in versions 1.0.1 and 1.0.2-beta.

If you have any questions or concerns, please don’t hesitate to contact our support team.

UPDATE: April 10, 2014, 11:08 PM PDT: Please note, that earlier today we re-issued our email certificates. To resolve any exception or error when it shows up regarding mismatch, you can try accepting the certificate. If that doesn’t help, you will need to remove your email address from your email client then re-add it and accept the security certificate/exceptions and server identities during that process.


12:14 pm

Scheduled Network Maintenance US-West (Irvine) Data Center Thursday April 10th 10 PM PDT (4hrs) (RESOLVED)

Posted (April 8th, 2014 at 12:14 pm PST) by andrea

On Thursday April 10th, starting at 10pm PDT, we will be performing network maintenance in our US-West (Irvine) Data Center. During the scheduled 4-hour window, we will be migrating the network layer from some older hardware to the newly designed setup. This is expected to be a near impact-free maintenance; all services will continue to work throughout the migration. However, you may experience momentary disruption when you have a service running on the specific portion of the network being moved. This includes anyone having any services in that data center as well as DreamObjects customers. As with any maintenance, we will follow a pre-tested process and will proceed slowly to ensure that we are able to address any issues that may occur.

Please note, we had to renumber a few apache web service IP addresses due to this maintenance. If you’re using our name servers, then this change is seamless for you; you don’t have to do anything. However, if you’re using outside name servers for your domain, you will have to update your A/Host record with your other provider if your current record set up with them matches any of the below:
208.113.144.3 changed to 208.113.145.57
208.113.148.2 changed to 208.113.148.193
64.90.60.2 changed to 64.90.60.89
67.205.12.3 changed to 67.205.13.22

Comments will be open so that we are aware if you’re seeing anything out of the ordinary. If you have any questions or concerns pre-maintenance, please contact support with the details, and we’ll be happy to help. You can check if your services are in the above data center in the DreamHost Panel.

UPDATE: April 11, 2014, 3:50 AM PDT: We are happy to announce maintenance has completed! Thanks once again for your patience at the extended maintenance window!

Previous updates available after the jump →


7:23 pm

Server downtime related to security updates (US-West: Los Angeles and Irvine, US-East) – RESOLVED

Posted (April 1st, 2014 at 7:23 pm PST) by Elizabeth

We are performing unplanned maintenance on multiple servers that may result in a very brief amount of downtime. A recent security update was released that caused a problem. This error did not compromise security but it made operations on a random number of servers across three data centers fail.

We are correcting the error and restarting the services of the affected machines. We do not currently have an estimated time this will be completed, other than as soon as possible.

We sincerely apologize for the unplanned nature of this issue.  If you have any questions please contact support.

UPDATE: April 1, 2014, 7:40 PDT:  All services have been restored, and we do not anticipate any further issues. If you do see any problem with your hosting, please contact support and ask us to check it for you.


12:03 pm

Mail Server Sending Issues Due to RBL Block

Posted (March 31st, 2014 at 12:03 pm PST) by Justin@DH

We are experiencing issues with customers ability to send mail to some mail providers, due to certain IP addresses belonging to our mail servers being listed on a RBL block. This can result in emails not being sent from your mail client, or emails being returned to their sender. Our abuse team is working with the entity blocking these ips, to get the issues resolved and the blocks removed as soon as possible.

We sincerely apologize to those customers who are being impacted by this issue and we will update this post when we have further details or news for you.

UPDATE: April 12, 2014, 10:50 PM PDT: We are happy to report that the Spamcop block situation has been under control for a little while now. We are continuing to keep an eye on anything that may cause the block again, and have recently shut down the major spammers. Even though we’re already very strict against outgoing spam, we’re also considering several system and policy enhancements to ensure continued unblocked status – not just with Spamcop but with other providers.

We are also looking into smaller blocks as they come up, so if you feel you’re affected, please provide us with a sample bounce error (dated the day you send it, not old ones, which may already have been resolved) with the full bounce email and headers of the original. Having this information is necessary to trace which IP or host is being blocked.

Previous updates available after the jump →


7:17 am

Web VPS Maintenance in US-West (Los Angeles) – April 1st 10PM PDT (2 hrs) (RESOLVED)

Posted (March 30th, 2014 at 7:17 am PST) by Justin@DH

On Tuesday, April 1st from 10PM to 12AM PDT, we will be performing maintenance on certain web VPS host machines in our US-West (Los Angeles) datacenter. To complete the process those host machines will need to be rebooted (which can take up to 20 minutes from start to finish). While this will result in brief service disruption, the reboot is necessary and will allow us to implement new features which will streamline future migration of web VPS services to different hardware without service interruption. In particular, this will allow us to replace older hardware newer and more efficient hardware (that change is in the works!). The following web VPS host machines will be affected:

homie-vserver36
homie-vserver38
homie-vserver39
homie-vserver40
homie-vserver41
homie-vserver49
homie-vserver50
homie-vserver51
homie-vserver52
homie-vserver54
homie-vserver189
homie-vserver394

You can verify which VPS host machine you are hosted on in the control panel data centers page. The details column lists the host machine for your VPS(s). If you have any questions please contact support.

We will update this post when the maintenance starts and with the progress. Thank you for your patience while we upgrade your services.

UPDATE: April 1, 2014, 22:57 PDT:  What? That’s it? Yes! Tonight’s maintenance has been completed. Please contact us (link above) if you find any issues with your services.

UPDATE: April 1, 2014, 22:15 PDT:  The maintenance has begun!

UPDATE: April 1, 2014, 22:05 PDT:  Just an update to let you know we’re just about ready to get underway with this! We will post another update when the maintenance begins.


11:20 am

Scheduled Maintenance for homiemail-sub1 (homiemail-master) email sub-cluster March 27th 7pm PDT (3hrs) (RESOLVED)

Posted (March 24th, 2014 at 11:20 am PST) by andrea

On Thursday, March 27th from 7PM-10PM PST, we will be performing maintenance on our “homiemail-sub1″ (homiemail-master) mail cluster. During this maintenance, we will be upgrading the load balancers for that mail sub-cluster to new software-based ones. This new solution has undergone rigorous testing and has been successful on three sub-clusters already, all of them larger than this one, so we’re confident it will make for a more effective handling of mail. Due to the nature of this maintenance, we do expect some service disruption during the window, but any e-mail sent during this time will be queued for delivery after the completion. This is the fourth time we’re doing this type of maintenance since January, so based on that experience, we were able to reduce disruption to the minimum.

To determine if your mail is handled on the above sub-cluster, please click on Account Status on the upper right corner in your DreamHost panel and look for the name under Your Email Cluster. It should state ‘homiemail-master’.

If you have any questions or need anything else please be sure to Contact Support. We will also be monitoring the comments on this post during the maintenance window.

Update March 27, 2014 8:08pm PST: This maintenance is now done! We are seeing no issues at this time and are resolving this maintenance. If you’re still seeing any problems with email, please let us know by contacting support.

Previous updates available after the jump →


5:31 pm

NS2 Hardware Upgrade – old IP address removed

Posted (March 18th, 2014 at 5:31 pm PST) by andrea

This is the follow up announcement we promised in our NS2.dreamhost.com Hardware Upgrade – IP Change Required for Vanity Name Servers post.

We are confident that this change will have only minimal effect because of the monitored and observed very low level of traffic through the old IP address (the change happened almost two months ago), but if you need assistance, please contact support with the details.


9:43 am

Web VPS Maintenance in US-West (Los Angeles) – March 18th 10PM PDT (2 hrs) (RESOLVED)

Posted (March 17th, 2014 at 9:43 am PST) by Justin@DH

On Tuesday, March 18th from 10PM to 12PM PDT, we will be performing maintenance on certain web VPS host machines in our US-West (Los Angeles) datacenter. To complete the process those host machines will need to be rebooted (which can take up to 20 minutes from start to finish). While this will result in brief service disruption, the reboot is necessary and will allow us to implement new features which will streamline future migration of web VPS services to different hardware without service interruption. In particular, this will allow us to replace older hardware newer and more efficient hardware (that change is in the works!). The following web VPS host machines will be affected:

homie-vserver57
homie-vserver58
homie-vserver59
homie-vserver60
homie-vserver61
homie-vserver62
homie-vserver64
homie-vserver65
homie-vserver66
homie-vserver67
homie-vserver69
homie-vserver292

You can verify which VPS host machine you are hosted on in the control panel data centers page. The details column lists the host machine for your VPS(s). If you have any questions please contact support.

We will update this post when the maintenance starts and with the progress. Thank you for your patience while we upgrade your services.

Update March 18, 2014 10:09pm PST:We are beginning the maintenance now! As noted above, there will be a short period of downtime when the VPS host machines are rebooted.

Update March 18, 2014 10:41pm PST:And like that, it’s done! If you see any issues with your service please contact support via the admin panel. Thanks!


3:21 pm

DreamObjects Maintenance March 18th 10AM PST (8 hrs) (RESOLVED)

Posted (March 11th, 2014 at 3:21 pm PST) by andrea

As part of our continued efforts to improve our cloud services, on Tuesday March 18th from 10am-6pm PST, our Systems Engineering team will be performing maintenance on our DreamObjects service. During this maintenance window, you may intermittently experience slower upload and access speeds and some short periods of connectivity loss. We will be upgrading to Ceph, the underlying storage technology that powers DreamObjects. The upgrade includes general performance improvements as well as additional features for managing the cluster.

While we anticipate increased latency when storing or retrieving objects during this maintenance, please rest assured that your data is safe and will not be affected. We will closely monitor the maintenance and will be posting periodic updates.

If you have any questions or concerns, please don’t hesitate to contact support for assistance.


 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
4:19 pm

Scheduled Maintenance for homiemail-sub3 email sub-cluster March 13th 7pm PST (4 hrs) (RESOLVED)

Posted (March 10th, 2014 at 4:19 pm PST) by andrea

On Thursday, Mach 13th from 7PM-11pm PST, we will be performing maintenance on our “homiemail-sub3″ mail cluster. During this maintenance, we will be upgrading the load balancers for that mail sub-cluster to new software-based ones. During this time, we will also be making backend DNS changes to our mail system. While this shouldn’t cause any customer-impacting issues, we do acknowledge that possibility and will keep a very close eye at all steps during the maintenance. This new solution has undergone rigorous testing and has proven to be effective already on the “homiemail-sub5″ and “homiemail-sub4″ subclusters, so we’re confident it will make for a more effective handling of mail. Due to the nature of this maintenance, we do expect some service disruption during the window, but any e-mail sent during this time will be queued for delivery after the completion.

To determine if your mail is handled on the above sub-cluster, please click on Account Status on the upper right corner in your DreamHost panel and look for the name under Your Email Cluster.

Please note: Once you have determined that you are on homiemail-sub3 cluster, and if you are using outside DNS (ie. you are not the default DreamHost nameservers), you have to update the IP address for your ‘mail’ A record to the new one. The new IP address is 69.163.253.7 .

If you have any questions or need anything else please be sure to Contact Support. We will also be monitoring the comments on this post during the maintenance window.

Update March 13, 2014 9:25pm PST: We consider this maintenance complete and successful. If you need any help with your mail service or anything else, please Contact Support with your details.

Previous updates available after the jump →


9:40 am

Power issues in our US-East Datacenter (Virginia) (RESOLVED)

Posted (March 10th, 2014 at 9:40 am PST) by Dan J

We had a power fluctuation in our US-East Datacenter around 9am PST today. This caused a large number of machines to reboot. We’re currently running on backup generators due to maintenance earlier this morning in this datacenter. All power has been restored now and appears to be stable, but we’re watching to make sure no other issues pop up.

Update March 10, 2014 10:30 am PST: It appears service has returned to normal now. We will continue to monitor the situation to make there are no lingering issues.  Thanks again for your patience. If you find that your site is not currently functioning as expected then please Contact Support.

Update March 10, 2014 10:24 am PST: Almost all machines are now online and functioning. We’re spot checking the last few unresolved problems.  Thanks again for your patience!

Update March 10, 2014 9:54 am PST: Most affected servers appear to have come back online correctly. Some servers however booted into an incorrect run level, which caused their hosted services to not start up properly. Our administrators are currently correcting those issues. Thank you for your patience!


4:41 pm

Scheduled Maintenance for homiemail-sub4 email sub-cluster February 27th 7pm PST (6 hrs) (RESOLVED)

Posted (February 24th, 2014 at 4:41 pm PST) by andrea

On Thursday, February 27th from 7PM-1AM PST, we will be performing maintenance on our “homiemail-sub4″ mail cluster. During this maintenance, we will be upgrading the load balancers for that mail sub-cluster to new software-based ones. During this time, we will also be making backend DNS changes to our mail system. While this shouldn’t cause any customer-impacting issues, we do acknowledge that possibility and will keep a very close eye at all steps during the maintenance. This new solution has undergone rigorous testing and has proven to be effective already on the “homiemail-sub5″ subcluster, so we’re confident it will make for a more effective handling of mail. Due to the nature of this maintenance, we do expect some service disruption during the window, but any e-mail sent during this time will be queued for delivery after the completion.

To determine if your mail is handled on the above sub-cluster, please click on Account Status on the upper right corner in your DreamHost panel and look for the name under Your Email Cluster.

Please note: Once you have determined that you are on homiemail-sub4 cluster, if you are using outside DNS (ie. you are not the default DreamHost nameservers), you will need to update the IP address for your ‘mail’ A record to the new one: 69.163.253.135.

If you have any questions or need anything else please be sure to Contact Support. We will also be monitoring the comments on this post during the maintenance window.

Update February 28, 2014 1:20 am PST: Everything looks great! Traffic is now resolving to the new load balancers, and these changes should result in significant stability improvements to our sub4 mail cluster. We are going to go ahead and close this maintenance tonight. As always, if you notice any issues with your mail services, please contact support! Thanks once again for your patience during this maintenance.

Previous updates available after the jump →


7:21 pm

Connectivity Issues with US-East (Virginia) Data Center (RESOLVED)

Posted (February 16th, 2014 at 7:21 pm PST) by Jen D.

We are seeing an increased amount of traffic to our East Coast data center and this is causing connectivity issues to your sites. Our systems and network engineering teams are currently investigating the spike in traffic and working to find the root cause.

We sincerely apologize to those customers who are being impacted by this issue and we will update this post as soon as we have any further news for you. If you are unsure if you are affected, you can check which data center you utilize in the control panel here.

Update 7:53pm PST: We’re not seeing any further connectivity issues within the last 10 to 15 minutes. We have estimated this attack to be a large 30-40 gbps. Since our mitigation efforts, we are not seeing any further issues at this time. We will be closing this post, but our teams will be watching for any further incidents closely through the night. We apologize for any inconvenience this has caused you.

Previous updates available after the jump →


7:55 pm

Network Maintenance US-West (Los Angeles) Data Center Thursday February 13th 10pm PST (5 hours) (RESOLVED)

Posted (February 11th, 2014 at 7:55 pm PST) by Brian@dreamhost

On Thursday February 13th, starting at 10pm PST, we will be conducting network maintenance in our US-West (Los Angeles) Data Center. The expected duration of this maintenance is 4 hours. During this maintenance, we will be upgrading the software on our core router. This upgrade requires a reboot of the core. In order to reduce any noticeable impact, we will be moving links, one by one, from the main core router to our temporary switches. During the moves, there will be mostly unnoticeable service impact, lasting no more than a couple of minutes. Once all links are moved, we will then reboot the core. This will be followed by moving the links back to the successfully rebooted core. Comments will be open during the maintenance so that we can assist you in real time, and verify if your issues are related to this maintenance.

You can check on the DreamHost Panel if you have services located in the US-West (Los Angeles) Data Center.
If you have any questions or concerns please contact the DreamHost Support Team.

Update February 14th 3:10am PST: All links have been moved back to the core and all services are reporting up without any issues. This maintenance is now completely done with no downtime at all. Thank you again for your patience throughout this maintenance. We are now resolving this status post successfully! Previous updates available after the jump →


7:01 pm

Connectivity Issues with US-East (Virginia) Data Center (RESOLVED)

Posted (February 11th, 2014 at 7:01 pm PST) by Elizabeth

We are experiencing unusually high traffic to our US-East data center. At this time, we are investigating the cause and we will let you know as soon as we have more information on what the problem is and what we are doing to alleviate it. We sincerely apologize to those customers who are being impacted by this issue. If you are unsure if you are affected, you can check which datacenter you utilize in the control panel here.

Update February 11th, 2014, 7:39 pm PST: The abusive traffic has been blocked and we are no longer experiencing trouble connecting with servers in US-East. If you are still seeing issues, please contact support and let us know so we can help you. Thank you for your patience.

Previous updates available after the jump →


7:54 pm

Live Chat Unavailable Wednesday, February 12th, 12am to Thursday, February 13th, 12am PST

Posted (February 7th, 2014 at 7:54 pm PST) by Brian@dreamhost

Our Live Chat support system will be unavailable this coming Wednesday, February 12th, from 12am to 12am (the following morning, 24hours). We will be involving our support staff in team meeting, discussing our 2014 plan, as well as specific changes to the teams scheduling and inner policies! We aim to offer the best service possible and we feel these changes will only help to improve our Live Chat team’s performance

Email support operations will still be available, should you need to get a hold of us. We will resume normal Live Chat operations at 12am PST the next morning, February 13th. We thank you for your patience and understanding.


3:41 pm

DreamObjects Maintenance February 11th 11AM-7PM PST (RESOLVED)

Posted (February 5th, 2014 at 3:41 pm PST) by andrea

As part of our continued efforts to improve our cloud services, our Systems Engineering team will be performing maintenance on our DreamObjects service on Tuesday February 11th from 11am-7pm PST. During this maintenance window you may intermittently experience slower upload and access speeds and short periods of connectivity loss. We will be upgrading Ceph, the underlying storage technology that powers DreamObjects. The upgrade will resolve an issue with bucket ACLs not getting saved and another related to large file uploads.

While we anticipate increased latency when storing or retrieving objects during this maintenance, please rest assured that your data is safe and will not be affected. We will closely monitor the maintenance and will be posting periodic updates.

If you have any questions or concerns, please don’t hesitate to contact the DreamHost Support Team for assistance.

Update February 11th, 2014, 12:50pm PST: The maintenance has been progressing as scheduled. As expected, we have been encountering 500 errors and sync disconnects as the upgrades roll out, for which we apologize. We’ll update this post again when we have more information.

Update February 11th, 2014, 3:50pm PST: We are happy to report that maintenance was successfully completed an hour ago. We are now resolving this status post, but if you have any questions or concerns, please don’t hesitate to contact the DreamHost Support Team for assistance.


5:40 pm

DreamHost Web Panel Currently Experiencing Connection Issues (RESOLVED)

Posted (January 29th, 2014 at 5:40 pm PST) by Elizabeth

We are seeing some instances of slow connections to the DreamHost web panel (panel.dreamhost.com). Our admins are investigating the root cause, and we will update here when we know more. We sincerely apologize for any inconvenience you may encounter.

Update January 29, 2014, 6:15 PM PST: At this time, all connectivity has been restored. Our administration team is continuing to investigate the root cause to ensure this doesn’t occur again.  If you are still seeing problems, please contact support and let us know! Previous updates available after the jump →


3:07 pm

NS2.dreamhost.com Hardware Upgrade – IP Change Required for Vanity Name Servers

Posted (January 28th, 2014 at 3:07 pm PST) by andrea

This is a notice to let you know that we are updating the hardware for one of our nameservers, ns2.dreamhost.com. We’re also moving it out of our California data center to the east coast data center to further diversify our nameserver locations. Due to the latter, the IP for ns2.dreamhost.com will be changed to 208.97.182.10 on February 5th, 2014.

Who will be affected by this IP change?

The only customers who will have to actively update their records are those who have their domain registered at another registrar *and* that domain is using glue records for vanity name servers that are pointed to our nameservers.

If you’re unsure if you’re using this setup, a simple WHOIS search will show you your current name servers and registrar.

If this does describe you, you can go ahead and log into that admin interface and make the changes with the new IP address listed above. Even though it’s not completely migrated yet, the new IP address is live and functional for your purposes. You can also just wait until the above date if you wish. There is no super rush to get this done, because after that change, the old IP will be functional for a while longer, but the sooner you make the change the better :)

We will post another announcement when we have a date for removing the old IP address, so keep an eye on dreamhoststatus.com if you can’t get to changing your records soon.

If you have your domain pointed directly to our nameservers, such as ns1.dreamhost.com, ns2.dreamhost.com (and ns3.dreamhost.com, which is optional), or you have vanity name name servers  on a domain that is currently registered with us, then this change does *not* affect you, as we automatically update everything in our system. If you’re unsure if you’re using this setup, again, a simple WHOIS search will show you your current name servers and registrar.

If you have any questions about this change, please contact the DreamHost Support Team.


 
© 1996-2014, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
3:44 pm

Network Maintenance, Switch Software Upgrades, US-West (Irvine) Data Center Thursday January 23rd 10pm PST (RESOLVED)

Posted (January 17th, 2014 at 3:44 pm PST) by Ralph

On Thursday January 23rd, starting at 10pm PST, we will be conducting network maintenance in our US-West (Irvine) Data Center. During the maintenance we will be upgrading the software on various switches throughout the Data Center. These upgrades will allow us to improve our monitoring capabilities in Irvine. After the software upgrades are complete, we will need to reboot the switches for the upgrades to be complete. These reboots will cause service downtime ranging from 2 to 10 minutes. Services affected will be FTP/SSH and access to your website. Mail will not be affected by this maintenance. We will proceed with these reboots slowly in the event that any hosting machine needs direct attention. This maintenance will mainly affect customers on dedicated server hardware. There is only one switch that is the home of shared and VPS customers, the rest are dedicated machines. We we will make every attempt to announce which machines are going to be affected as we progress through the maintenance, so please keep an eye out for our updates. When the maintenance starts, comments will be open so that we can assist with you any possible issues that may come up after the reboots.

You can check on the DreamHost Panel if you have services located in the US-West (Irvine) Data Center. During the maintenance, we will do our best to let you know exactly which machines will be affected and when.

If you have any questions or concerns please contact the DreamHost Support Team.

Update January 24th 12:40pm PST: All 9 public reboots and the 6 private network reboots have been completed successfully with very low impact on server availability. Thank you again for your patience throughout this maintenance. We are now resolving this status post successfully!

Previous updates available after the jump →


11:32 am

Network Improvement Series Maintenance US-West (Los Angeles) Data Center Thursday January 16th 10pm (4 hour) (RESOLVED)

Posted (January 15th, 2014 at 11:32 am PST) by Ralph

As some of you know, we have been working very hard to improve our network over the past 2+ years with our Network Improvement Series. On Thursday January 16th, in a 4 hour window starting at 10pm, we will be moving on to the next step of the Network Improvement Series maintenances. This step is going to involve migrating the network layer from the old hardware to the new designed setup. This is expected to be a near impact-free maintenance; all services will continue to work throughout the migration. As with all maintenances, we will proceed slowly to ensure that, should any problems occur, we will be able to address them quickly. Comments will be open so that we can address any problems you may be experiencing. We are excited to continue and eventually conclude the major network overhaul we’ve been working on to improve your hosting experience. We thank you for your patience through the years.

You can check on the DreamHost Panel if you have services located in the US-West (Los Angeles) Data Center.

If you have any questions or concerns please contact the DreamHost Support Team.

Update January 17th 1:48am PST: We’re happy to report that this step of our network maintenance series is now complete. We are now on the new hardware, and the process was completed without any customer impact. Keep an eye out for upcoming maintenance announcements, and as always if you have any questions or concerns, please contact the DreamHost Support Team.

Previous updates available after the jump →


7:01 pm

Network Maintenance US-West (Irvine) Data Center Friday January 10th @ 10:45pm PST (RESOLVED)

Posted (January 10th, 2014 at 7:01 pm PST) by Sean

Due to the recent connectivity issues experienced on January 7th in the US-West (Irvine) data center and some changes made in our related maintenance last night, our team has decided it is best to conduct another maintenance as soon as possible on our network gear to help prevent future connectivity issues that could lead to downtime. Part of our maintenance plan includes moving links to temporary switches, then back again to the core. When working with the network, even with redundant distribution, there is always the risk of some service impact. Due to this risk, we will be monitoring the network, comments and support requests to ensure that we keep you informed and are able to adjust our processes throughout the entirety of the maintenance window.

You can check on the DreamHost Panel if you have services located in the US-West (Irvine) Data Center.

If you have any questions or concerns please contact the DreamHost Support Team.

Update January 11th 02:12am PST: The network maintenance has been completed!

Update January 11th 02:12am PST: We are moving all links back to the core from the temp switches. So far so good.

Update January 11th 01:50am PST: Everything is looking real good at the moment. IPv6 connectivity has also been restored. The maintenance is still moving along and we’ll provide another update soon.

Update January 11th 01:22am PST: The network maintenance is still ongoing. We have not seen any issues but are continuing to monitor everything closely.

Update January 11th 01:00am PST: Changes being made right now will impact account changes in the admin panel. Otherwise, you should not see any issues with your service. We should be finished with this part soon so changes will no longer be delayed.

Update January 11th 12:43am PST: It’s been smooth sailing for this network maintenance. This is just an update to let you know everything is still going well.

Update January 11th 12:18am PST: Everything is moving along smoothly as we continue to move links to the temporary switches. No services have been interrupted.

Update January 10th 11:42pm PST: The network maintenance is progressing as planned. We are currently moving links over to the temp. switches. Everything is looking good and we have not found any interruption of services.

Update January 10th 11:17pm PST: We are now beginning the network maintenance. There should not be any interruption of services.

Update January 10th 10:50pm PST: We are getting ready to begin this maintenance soon.


10:59 am

Emergency Network Maintenance US-West (Irvine) Data Center Thursday January 9th 10pm PST (RESOLVED)

Posted (January 9th, 2014 at 10:59 am PST) by Ralph

Due to the connectivity issues experienced on January 7th in the US-West (Irvine) Data Center, our team has decided it is best to conduct maintenance on our routing network gear to help clear up a lingering issue with IPv6 accessibility. Our maintenance plan will enable us to keep all services active while we work towards correcting IPv6 support at the US-West (Irvine) location. When working with the network, even with redundant distribution, there is always the risk of some service impact. Due to this risk, we will be monitoring the network, comments and support requests to ensure that we keep you informed and are able to adjust our processes throughout the entirety of the maintenance window.

You can check on the DreamHost Panel if you have services located in the US-West (Irvine) Data Center.

If you have any questions or concerns please contact the DreamHost Support Team.

Update January 10th 1:20am PST: This concludes tonights maintenance. We will have another announcement once we are ready to attempt our next maintenance to correct IPv6 issues in US-West (Irvine).

Previous updates available after the jump →


2:50 pm

Live Chat to be turned off for short internal training sessions on January 9th and January 15th (RESOLVED)

Posted (January 8th, 2014 at 2:50 pm PST) by andrea

In order to further improve our Live Chat customer experience, we are conducting training sessions for our team members on 1/9 at 12pm noon and at 4pm PST and on 1/15 12:15am PST. These are short, up to one hour, sessions each but we deemed it necessary to turn off Live Chat for the duration. You will still be able to contact our tech support team by submitting a ticket. We apologize for the inconvenience.


7:18 pm

Connectivity issues in US-West (Irvine) Datacenter (RESOLVED)

Posted (January 7th, 2014 at 7:18 pm PST) by Elizabeth

We are currently experiencing issues with a small subset of servers in our US-West (Irvine) data center. At this time, we are investigating the cause and we will let you know as soon as we have more information what the problem is and what we are doing to alleviate it. We sincerely apologize to those customers who are being affected by this issue. If you are unsure if you are affected, you can check which datacenter you utilize in the control panel here.

Update: January 7, 2014 10:11pm PST: At this time, all connectivity has been restored and all internal alerts are back to normal. We are resolving this issue for now and will continue to investigate the root cause to ensure this doesn’t pop up again.  If you are still seeing problems, please contact support and let us know!

Previous updates available after the jump →


4:12 pm

Scheduled Maintenance for homiemail-sub5 email sub-cluster January 2nd 7:30pm PST (2.5hr) (RESOLVED)

Posted (December 30th, 2013 at 4:12 pm PST) by andrea

On Thursday, January 2nd from 7:30PM-10PM PST, we will be performing maintenance on our “homiemail-sub5″ mail cluster. During this maintenance, we will be upgrading the load balancers for that mail sub-cluster to new software-based ones. This new solution has undergone rigorous testing and we’re confident it will make for a more effective handling of mail on that sub-cluster. Due to the nature of this maintenance, we do expect some service disruption during the window, but any e-mail sent during this time will be queued for delivery after the completion.

To determine if your mail is handled on the above sub-cluster, please click on Account Status on the upper right corner in your DreamHost panel and look for the name under Your Email Cluster.

If you have any questions or need anything else please be sure to Contact Support. We will also be monitoring the comments on this post during the maintenance window.

Update: January 2nd, 8:50pm PST: We are happy to report that this maintenance is now complete. The homiemail-sub5 email cluster is now being managed by our new software-based load balancer, with a secondary balancer standing by for redundancy. All tests and re-checks came back as expected, and we’re resolving this status post. If you have any questions or concerns, please be sure to contact support with details.

Previous updates available after the jump →


3:15 pm

Limited number of servers in US-West (Los Angeles) experiencing connectivity problems (RESOLVED)

Posted (December 28th, 2013 at 3:15 pm PST) by Elizabeth

We are currently experiencing issues with a small subset of servers in our US-West (Los Angeles) data center. At this time, we are investigating the cause and we will let you know as soon as we have more information what the problem is and what we are doing to alleviate it. We sincerely apologize to those customers who are being affected by this issue. If you are unsure if you are affected, you can check which datacenter you utilize in the control panel here.

Update: December 28th, 4:07 PM PST: The connectivity issues have been resolved. The root cause of the issue was a fuse that needed to be reset on the power strip for that rack. All power has been restored and affected machines are now back online. Should you have any further issue with your sites, please contact support and we will be glad to look into the issue for you!

Previous updates available after the jump →


8:25 pm

Connectivity Issue in one row of US-West (Irvine) Data Center (RESOLVED)

Posted (December 18th, 2013 at 8:25 pm PST) by Elizabeth

We are currently experiencing issues with a small subset of servers in our US-West (Irvine) data center. At this time, we are investigating the cause and we will let you know as soon as we have more information what the problem is and what we are doing to alleviate it. We sincerely apologize to those customers who are being affected by this issue. If you are unsure if you are affected, you can check which datacenter you utilize in the control panel here.

Update December 18th, 8:56 pm PST: We have mitigated the abusive traffic and there should no longer be issues on any server in Irvine. If you are experiencing any problems, please contact support and let us know! Thank you for your patience!

Previous updates available after the jump →


10:12 pm

DreamObjects – Trouble With Large Objects (RESOLVED)

Posted (December 17th, 2013 at 10:12 pm PST) by Steven

Following today’s update of Ceph, the software that powers DreamObjects, we have been seeing issues with large objects (> 500k in size). Troubleshooting began during scheduled maintenance. Our DreamObjects engineers began looking into the problem, identified it, and a fix is forthcoming. We apologize for any inconvenience this may cause and thank you for your patience! Please check back for updates as we will be posting them as soon as they become available.

This applies to DreamObjects only, no other services are affected. If you have any questions or need anything else please be sure to Contact Support.

Update December 18th, 12:28am PDT The fix has been implemented and calling this resolved! We have not found any other issues with our testing but please do not hesitate to Contact Us. Thanks again for your patience!

Previous updates available after the jump →


 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
10:31 am

WordPress 3.8 Major Release Now Available

Posted (December 12th, 2013 at 10:31 am PST) by Mika E.

First it was WordPress 3.6 in August and then WordPress 3.7 in October, and now it’s December and surprise, it’s WordPress 3.8!

The newest version of WordPress (3.8 aka “Parker” after Charlie Parker), is now available on the DreamHost panel for new installs and upgrades. You can also upgrade from directly within your site’s WordPress admin panel, if you prefer.

While 3.7 was a sneaky release, trust me, you’ll notice 3.8. The dashboard had an overhaul, with new colors for everyone’s preferences (no more just blue and grey!), and the whole design is modern and responsive. That means all your smart phones and tablets will be able to use WordPress with even greater ease than ever before! The theme dashboard got a facelift too, as did widgets, so you’ll find changing your site’s design to be far easier.

Now, we understand the update fatigue. A new major release every 2 months is tough (especially on us). We’ve got your back, though! Over the next few days, this update will be automatically handled for all customers who have automatic upgrades enabled, including all those on DreamPress. If you used our One-Click Installer to install WordPress, you can check your settings, and change these preferences directly, by visiting the DreamHost One-Click Panel. Once there, click “Manage Installed Applications,” select the domain, then look under the Upgrade Action column for details.

If you run into issues after your upgrade, as a first step, take a look at this thread on 3.8 in the DreamHost forums. Unlike previous releases, there is not (at this point) any plan to make an official ‘issue tracking’ thread on WordPress.org forums, but please do go there to report any problems you have with specific plugins and themes. Every single plugin developer was contracted, prior to release, to remind them to test, so we’re hoping that worked. It was a very large email list.

Remember, only minor releases (like 3.8 to 3.8.1) will now be automatically applied by WordPress. We’ll still run our One-Click updates for everyone, just in case the automatic tool doesn’t work (or is disabled). But you’d never disable that, right?


6:03 pm

Mail Upgrade for Homiemail-Sub1, Wednesday December 11th 6pm – 10pm(POSTPONED)

Posted (December 10th, 2013 at 6:03 pm PST) by Ralph

On Wednesday, December 11th from 6PM-10PM PST, we will be performing maintenance on our “homiemail-sub1″ mail cluster. We will be using this time to migrate customers on this cluster to another set of load balancers. We expect minimal service disruption during this maintenance, and any e-mail sent will be queued for delivery after it’s completion.

This only affects any users on our homiemail-sub1 mail cluster. To check if you are affected, click the “Account Status” link on the top right of your DreamHost panel.

If you have any questions or concerns, please contact our tech support team, and we’ll be happy to help.

Update December 11th 3:00PM PDT: We have unfortunately decided that it would be best to postpone this maintenance once more. As we near any maintenance, we are constantly evaluating our plan and monitoring service to make absolutely sure that our maintenance will not only run as smoothly as possible to prevent any further service interruptions, but also address and correct the problems we’re experiencing at this time. We will be revisiting this maintenance in the near future, once we are confident that we can complete it successfully.


4:52 pm

DreamObjects Scheduled Maintenance – December 17th, 9am PDT – 19th, 5pm PDT (RESOLVED)

Posted (December 6th, 2013 at 4:52 pm PST) by andrea

Our Systems Engineering team will be performing maintenance on our DreamObjects service from Tuesday December 17th through Thursday December 19th. During this maintenance period, you may intermittently experience degraded performance. The purpose of this maintenance is to implement key upgrades to Ceph, the underlying storage system which powers DreamObjects. As a result of the upgrades, we’ll be able to provide some exciting new features, including CORS support and improved multi-part uploads. While we anticipate increased latency when storing or retrieving objects, please rest assured that your data is safe and will not be affected.

If you have any questions or concerns about the upcoming maintenance, please contact our support team.

Update December 20th, 11:40am PDT Just a quick note to let you know that as previously mentioned, the upgrade of this service has been successful. We resolved the post yesterday, but please let us know by href=”https://panel.dreamhost.com/index.cgi?tree=support.msg&” target=”_blank”>contacting our support team if there is anything else we can do for you.

Previous updates available after the jump →


1:52 pm

Stability Issues Impacting a Small Number of Hosting Servers in US-West (Los Angeles) (RESOLVED)

Posted (December 6th, 2013 at 1:52 pm PST) by ScottK

Memory issues are causing a subset of servers on our US-West (Los Angeles) data center to become unresponsive. We are seeing similar behavior in a couple of other data centers as well but on a much smaller scale. We are working with our network, admin and security teams to determine the cause and fix for this specific issue. We will provide additional information and servers affected when available.

Update December 5th, 14:48 PDT – The server stability issue has been resolved. Our team is continuing to investigate the matter to ensure that it does not occur again. Thank you again for your patience.

Update December 5th, 14:13 PDT – Improvements to server stability have been made and alerts are continually dropping. We are still investigating the root cause and will update this status post with more information as soon as possible.

Update December 5th, 14:06 PDT – We are currently working on all affected servers individually and recovering services on them. We have our entire team addressing this issue.  Thank you for your patience.



11:39 am

Stability Issues Impacting a Small Number of Hosting Servers (RESOLVED)

Posted (December 5th, 2013 at 11:39 am PST) by Justin@DH

Memory issues are causing a subset of servers on our US-West data center to become unresponsive. We are seeing similar behavior in a couple of other data centers as well but on a much smaller scale. We are working with our network, admin and security teams to determine the cause and fix for this specific issue.  We will provide additional information and servers affected when available.

Update December 5th, 12:03pm PDT: Improvements to server stability have been made and we are seeing less issues that could affect customers. We are continuing our efforts to resolve the remaining issues.

Update December 5th, 12:40pm PDT: Our monitoring system alerts have dropped significantly at this time. We aren’t resolving this post yet to sure the root case is found and fixed. When we have a final resolution or other chance of situation we will update this post.

Update December 5th, 1:03pm PDT: We are examining a large amount of anomalous network traffic that we think is related. We will post further details as we have them. We are resolving the post but are continuing to investigate and monitor the situation closely.


11:20 am

DreamObjects currently experiencing service interruption (RESOLVED)

Posted (December 5th, 2013 at 11:20 am PST) by Ralph

DreamObjects is currently experiencing a service interruption. We first noticed the situation around 10:53am PST. Our team is currently investigating this unexpected downtime. More details to come. No other services are affected by this downtime, only DreamObjects. We will update this status post as we find out more about this situation. At this time we do not currently have an ETA for return of service. Apologies for the interruption, we will do our best to get the service running as soon as possible. Your patience is greatly appreciated.

Update December 5th 11:20am PST: DreamObjects is now working correctly. We have determined the load balancer to be the cause of the issue. Our team will continue investigation to make sure that this type of problem can not come up again. Apologies for the unexpected issues.


1:34 pm

Connectivity Issue in US-West (Irvine) Data Center (RESOLVED)

Posted (November 25th, 2013 at 1:34 pm PST) by Sean

We are currently experiencing a connectivity issue in our US-West (Irvine) data center. We’re investigating the cause and are working towards a resolution. We apologize for the inconvenience this is causing, and will post updates as the work progresses. Web servers, mysql and email may be affected by this issue. Thanks for your patience!

Update November 25th 1:55PM PDT: All hosts are up and responsive, we’re in the clear. If you’re experiencing any issues, please contact support! Thanks again for your patience.

Previous updates available after the jump →


2:05 pm

Mail upgrade for homiemail-sub1 cluster, Dec 2nd 1AM-2AM (1 hour maintenance) (RESOLVED)

Posted (November 21st, 2013 at 2:05 pm PST) by Seohee

On Monday, December 2nd from 1AM-2AM PST, we will be performing maintenance on our “homiemail-sub1″ mail cluster. We will be using this time to migrate customers on this cluster to another set of load balancers, which should help alleviate recent issues. We expect minimal service disruption during this maintenance, and any e-mail sent will be queued for delivery after it’s completion.

This only affects any users on our homiemail-sub1 mail cluster. To check if you are affected, click the “Account Status” link on the top right of your DreamHost panel.

If you have any questions or concerns, please contact our tech support team, and we’ll be happy to help.

Update December 1st 1:30PM PDT: Unfortunately, we ran in to some issues with the migration and we will not be ready to perform this upgrade at the scheduled time. We will be rescheduling it, but don’t have a date or time for it yet. We will create a new status post announcement once we have scheduled a new timeframe. We’re sorry for any inconvenience this causes you.


3:22 pm

Mail upgrade for homiemail-sub1 cluster, Nov 25th 1AM-2AM (1 hour maintenance) POSTPONED

Posted (November 20th, 2013 at 3:22 pm PST) by Brandon

On Monday, November 25th from 1AM-2AM PST, we will be performing maintenance on our “homiemail-sub1″ mail cluster. We will be using this time to migrate customers on this cluster to another set of load balancers, which should help alleviate recent issues. We expect minimal service disruption during this maintenance, and any e-mail sent will be queued for delivery after it’s completion.

This only affects any users on our homiemail-sub1 mail cluster. To check if you are affected, click the “Account Status” link on the top right of your DreamHost panel.

If you have any questions or concerns, please contact our tech support team, and we’ll be happy to help.

Update Nov. 21st 2:45pm PST: This maintenance has been temporarily postponed to December 2nd. New status post has been created. We have determined that our plan for the upgrade needed some further changes to ensure that the upgrade will go as smoothly as possible and provide a superior level of service once complete.


7:15 am

Mail Issues with homiemail-sub3 and homiemail-sub4 (RESOLVED)

Posted (November 20th, 2013 at 7:15 am PST) by Justin@DH

We are experiencing issues with a specific file server that has lost connectivity to the network.  This is causing service degradation for servers in the listed mail clusters which has the potential to disrupt email access (both receiving and sending).  Members of every team are working on the issue at this time and hope to have it resolved soon. We will provide more information as it becomes available.

If you are unsure if you are impacted by this issue please check the account status drop-down on the top right of the account management panel . Your email cluster is listed at the bottom.

Update November 20th 7:30am PDT:Connectivity to the file server has been restored by our teams.  The mail servers using that machine will need to catch up on all the data they attempting to write to it and so may be slower than normal for a short period of time. Just to be safe, we are monitoring these servers to make sure they recover (we anticipate normal mail service will be restored shortly once everything is caught up).

Update November 20th 7:55am PDT:Connectivity is still looking fine. The mail machines are still recovering from the file server issue, but are nearly back to normal. There may be some delays on receiving messages until all the queues on the mail machines have caught up. We are continuing to monitor these servers to make sure they recover (we anticipate normal mail service will be restored shortly).


 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
2:19 pm

Network Maintenance Affecting All Data Centers Nov. 19th 10pm – Nov. 20th 2am PST(4 hours) (RESOLVED)

Posted (November 15th, 2013 at 2:19 pm PST) by Ralph

On Tuesday, November 19th, starting at 10pm PST we will be performing network maintenance in our three data centers:

US-West (Los Angeles), US-West (Irvine) and US-East (Virginia)

This maintenance will improve communication between our three data centers.

The changes will require propagation across the internet via the BGP routing protocol. As a result, your service may appear to be down during the maintenance for a short period that should not exceed 5 minutes. Service interruption will largely depend on your ISP and how quickly the changes register within their network. To help identify any problems, we will have our staff monitoring the changes from multiple locations and ISPs.

We will be performing this maintenance in three stages with one for each data center. We will start with our US-West (Los Angeles), followed by US-West (Irvine) and finish up with US-East (Virginia). We will provide updates as we move through the maintenance.

To see which data center your site is hosted in, please look at Status->Data Centers section in the panel.

Comments will be opened when the maintenance starts so that we can assist you with any issues you may be noticing.

If you have any questions about the upcoming maintenance, please contact our support team.

Update November 20th 1:50am PDT: That last update went perfectly! We are now in finished, and can close out this maintenance window. We appreciate everyone’s patience and are happy to report no issues. If you see your site having problems, please contact our support team and let them know!

Update November 20th 1:40am PDT: We’re nearing completion, but this last step could cause some downtime for our US-East (Virginia) datacenter. We will update when this step is complete. We appreciate your patience!

Previous updates available after the jump →


5:26 pm

Future PHP upgrades – your input is wanted!

Posted (November 14th, 2013 at 5:26 pm PST) by andrea

As you may have noticed, we now offer three version of PHP on our web servers. 5.2 is really on its way out and isn’t recommended any more. You can read about it here.  The version most likely to work with common web applications  is 5.3, but we did make 5.4 available for those of you who wanted to move ahead with that. Please note that with any version of PHP, you may be noticing incremental upgrades and patches from time to time. If your domain is set to run 5.4, these upgrades may or may not affect you. If they do affect you, you will have to either remove/upgrade the affected application or downgrade your php version for that domain in the DreamHost panel to 5.3. (edit your domain and scroll down to Web Options) That said…

Due to some incompatibility concerns that arose since the last time we upgraded to the latest PHP version, we are starting a conversation with you, the customer, to gain better insight and to provide a forum for you to contribute your findings, ideas and questions and concerns.

We welcome you to share your thoughts over in the PHP Versions: Upgrades, Observations, and Troubleshooting thread.

We will do our best to post all version upgrades (incremental or full), and will be monitoring the thread for your feedback. We would like to hear from you: what did you notice, did it affect your site, what exactly was affected, etc. We are hoping this to become not just an information exchange among our customers, but that we can also learn from it to better prepare for our upcoming upgrades. We do the upgrades to continue protecting the servers your sites are hosted on, but we understand that there is always room for improvement!


3:21 pm

WordPress Theme Vulnerability: Themify

Posted (November 13th, 2013 at 3:21 pm PST) by Mika E.

This morning, Themify announced that there was a security vulnerability in their theme. This vulnerability was caused, in part, by their own upgrade process which did not delete a vulnerable file. As a result of this announcement this morning, we, DreamHost, blocked the file from being able to be used to hack into sites.

It’s important to note that the fix we applied only works if you have Mod Security enabled. Most of you should, it’s default in our Panel. But if you, for some reason, turned it off, we really think you oughta go back and make sure that “Extra Web Security?” is checked in your domain’s edit page.

While this means your sites are safe, if you’re using Themify we strongly suggest you reinstall their theme template manually. You’ll want to download a fresh version of their framework, delete the themify folder (NOT your child themes, don’t panic, this won’t break your site’s layout), and upload the new one. You can’t just overlay the files, sadly, you really have to delete the old folder for this to be fully patched.

For more information, please read the Themify Blog — URGENT: Vulnerability Found in Themify Framework, Please Read.

Please contact our tech support team if you have any questions regarding this announcement.


2:44 pm

Service change notifications delayed (RESOLVED)

Posted (November 8th, 2013 at 2:44 pm PST) by andrea

Due to a hardware failure, service notifications generated by our DreamHost panel are temporarily delayed. Please note, the service changes you make in the panel are are going through, it’s just the confirmation messages of those changes, like creating an email address, adding a domain, changing an FTP user, essentially all the service management you can do on the DreamHost panel, that are taking a bit longer than usual.
We are working on getting this situation remedied, and appreciate your patience in the meantime. Please check back for updates.

Update November 8th 3:05pm PDT: We are happy to report that the service change notifications are back on track. Please accept our apologies for any inconvenience their delay may have caused.


12:28 am

Central database maintenance – Friday November 8, 3:45am PDT (RESOLVED)

Posted (November 8th, 2013 at 12:28 am PST) by chalise

At 3:45AM PDT, we will be performing maintenance on our central database. The DreamHost panel (https://panel.dreamhost.c0m) will be inaccessible during this time, but will not cause any downtime for your sites, email, databases, etc. This only affects the panel itself. We expect the maintenance to be done by no later than 4:45am PDT.

We apologize for the lack of advanced warning and will update this post again once the maintenance begins. Thanks for your understanding!

UPDATE Nov 8, 2013; 4:38am PDT: Maintenance has now completed and the panel is accessible once again! Thanks once again for your patience and understanding!

Previous updates available after the jump →


4:09 pm

DreamObjects Maintenance – Wednesday November 6th 10am PDT (8hrs) (POSTPONED)

Posted (November 1st, 2013 at 4:09 pm PST) by andrea

This maintenance is the second part of DreamObjects Maintenance – Wednesday October 30th 10am PDT

As we were not able to complete the expected improvements, we need to continue and complete this work on Wednesday, November 6th from 10 AM PST to 6 PM PST. We do not anticipate service interruption (the issues that came up previously have been completely resolved) but we provide the notice and window so that you are aware of what’s going on.

If you have any questions or concerns, please contact our tech support team, and we’ll be happy to help.

Update November 6th 6:09pm PST: Unfortunately, we ran into some difficulties while trying to complete the maintenance. They weren’t at the point of affecting your service availability, but we didn’t want to take any chances. We will be re-evaluating our process and will re-attempt the maintenance at a future date. We will add a status post as soon as we know the date and you will also receive an email if you have DreamObjects service with us. Sorry about any inconvenience this may have caused.

Previous updates available after the jump →


4:12 am

UPS Maintenance: US-East (Virginia) Datacenter (RESOLVED)

Posted (October 31st, 2013 at 4:12 am PST) by chalise

We are beginning our scheduled annual maintenance on our UPSes in our US-East (Virginia) Datacenter.

Revised maintenance windows:
November 1st, 5 AM – 7 AM PST
November 1st, 7 AM – 2 PM PST
November 4th, 4 AM – 2 PM PST
November 5th, 4 AM – 2 PM PST

This is not expected to cause any downtime, however, we will keep this post updated with major developments as the maintenance progresses.

Update Nov 5 2:47pm PST: The final phase of our maintenance is complete. As expected this did not have any negative impact on customer sites.

Previous updates available after the jump →


7:10 pm

Live Chat Unavailable Monday October 26th 8pm – October 27th 10am PDT

Posted (October 25th, 2013 at 7:10 pm PST) by Brian@dreamhost

Our Live Chat support system will be unavailable this coming Saturday, October 26th, from 8pm to 10am PDT (the following morning). We will be involving our support staff in our annual Halloween party, rewarding them for their excellent service and hard work! We aim to offer the best service possible and we feel our team deserves it!

Email support operations will still be available, should you need to get a hold of us. We will resume normal Live Chat operations at 10am PDT the next morning, Oct 27th. We thank you for your patience and understanding.


1:22 pm

DreamObjects Maintenance – Wednesday October 30th 10am PDT (6hr)(POSTPONED)

Posted (October 25th, 2013 at 1:22 pm PST) by andrea

On Wednesday, October 30th from 10AM-6PM PDT, we will be performing maintenance on the network that DreamObjects uses. The maintenance work includes adjusting our address scheme to a more resilient design, so that we can reduce potential points of failure. We are not expecting any customer-facing downtime during the maintenance, regardless, our team will be closely monitoring the process.

If you have any questions or concerns, please contact our tech support team, and we’ll be happy to help.

==================
About DreamObjects
DreamObjects is DreamHost’s object storage service. Object storage is like a giant hard drive on the internet that can never fill up. It’s designed to store static content, so it’s great for images, videos, application data, logs, etc. There’s even a WordPress plug-in to backup your site to DreamObjects! Store it reliably and cost-effectively with DreamObjects. Learn more at DreamObjects.

Update October 30th 4:00pm PDT: We are confident that we are no longer experiencing any issues caused by today’s maintenance. We will need to postpone the remainder of this maintenance to next week. Our confidence is high that we can perform next week’s maintenance with largely unnoticed impact. We will however conduct the maintenance in a window, in case anything unexpected should occur. We sincerely apologize for any inconveniences this has caused and we thank you for your continued patience. When we are confident to move forward next week, we will be providing direct email contact, as well as another status post, to detail our next attempt.

Previous updates available after the jump →


4:28 pm

WordPress 3.7 Major Release Now Available

Posted (October 24th, 2013 at 4:28 pm PST) by Mika E.

It’s only been a couple months since 3.6, I know, but here we go again!

The newest version of WordPress (3.7 aka “Basie”), is now available on the DreamHost panel for new installs and upgrades. You can also upgrade from directly within your site’s WordPress admin panel, if you prefer.

3.7 is a sneaky release, most of the cool things are behind the scenes so you may not actually notice a huge change. That’s okay!

For more details, you can check out the release notes here, with further details available in the WordPress codex. Heck you should check the writeup on poststat.us too!

If you run into issues after your upgrade, as a first step, take a look at this thread on 3.7 in the DreamHost forums, and the 3.7 master list at the WordPress.org forums, where issues relating to upgrades are being tracked and updated as they come up.

Over the next few days, this update will be automatically handled for all customers who have automatic upgrades enabled, including all those on DreamPress. If you used our One-Click Installer to install WordPress, you can check your settings, and change these preferences directly, by visiting the DreamHost One-Click Panel. Once there, click “Manage Installed Applications,” select the domain, then look under the Upgrade Action column for details.

Oh and in the manner of Steve Jobs, one more thing. Minor releases (like 3.7 to 3.7.1) will now be automatically applied by WordPress. We won’t be disabling this as we have absolute faith in this feature. That said, we’ll still run our One-Click updates for everyone, just in case the automatic tool doesn’t work (or is disabled). But you’d never disable that, right?


 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
10:49 am

Network Maintenance US-West Data Center (Los Angeles) Core Router Configuration Update – Wednesday October 23rd 10pm PDT (RESOLVED)

Posted (October 22nd, 2013 at 10:49 am PST) by Ralph

On Wednesday October 23rd starting at 10pm PDT, we will be conducting maintenance on our core routers in our US-West Data Center (Los Angeles). The window for this maintenance will last for 4 hours, till 2am PDT, Oct. 24th. During this maintenance, we will be reconfiguring our two core routers to optimize support for IPv6. Our maintenance plan will allow us to reconfigure our routers without causing significant impact on the services we provide. We will also allow for time to monitor the system to ensure that there are no unexpected issues.

The overall expected impact of this maintenance should be unnoticeable, however, we will open up comments and monitor support throughout the window.

Please contact support if you have any questions. To find out if you may be affected by this maintenance, go to the DreamHost panel to check the location of your services. Comments will be turned on during the maintenance to assist you in real time, should we experience any unexpected difficulties.

Update October 24th 12:00am PDT: This maintenance is now complete. Thank you for your patience. We are happy to announce this maintenance was completed with no noticeable impact.

Previous updates available after the jump →


12:43 pm

Upgrading from PHP 5.2

Posted (October 16th, 2013 at 12:43 pm PST) by mir

PHP 5.2 has been defunct for some time (it was officially moved to ‘unsupported’ January 6, 2011) and thus is very outdated. In an effort to help secure your sites and data we’ve made it easier than ever to upgrade your PHP version by adding a feature to the panel here:

https://panel.dreamhost.com/index.cgi?tree=home.over&

which will identify domains which have PHP 5.2 running and let you upgrade them with just a couple of clicks. Please take the time to check this out and make sure you are up-to-date. You may also want to read this article if you have questions about upgrading:

http://php.net/manual/en/migration53.php

Please note that it is still possible to revert to PHP 5.2 at this time if it causes any issues with your site (you may just need to upgrade your applications which is also very important for sound security practices).


3:33 pm

Alert: NextGenGallery v2.0.23

Posted (October 14th, 2013 at 3:33 pm PST) by Mika E.

Last month, NextGEN Gallery pushed a release that had a flaw in how it handled temporary data, causing users to suffer database issues. If you’re still using version 2.0.23 of the plugin, we urge you to upgrade at the earliest as this will correct the issue and clean out the database issues as needed. NextGEN Gallery worked closely with core WordPress developers to resolve this issue, and are aware the 2.0 upgrade hasn’t been at all as smooth as they would have liked.

For more information on the recent changes, see the NextGEN Gallery Changelog or visit their blog to read about the recent changes.

You can upgrade NextGEN Gallery via your WordPress Dashboard.


2:43 pm

Network Maintenance – Router Redundancy Improvements and Data Center Maintenance – Monday Oct. 14th 10pm US-West (Los Angeles) (RESOLVED)

Posted (October 11th, 2013 at 2:43 pm PST) by Ralph

This Monday Oct. 14th starting at 10pm, we will be conducting maintenance to reestablish redundancy, which was impacted by a hardware failure last week. During this maintenance we are expecting to see a few 30 second interruptions of service. These network interruptions should be unnoticeable.

Once the router maintenance is complete, at around 12 midnight, our data center will then be conducting another maintenance. This maintenance is similar to Thursday’s maintenance, which was completed successfully in roughly 30 minutes. We will not be conducting this part of the maintenance, but we will have our team on hand to make sure everything goes smoothly, as this data center maintenance does affect our network. The data center maintenance is expected to have no impact on the services we offer.

Please contact support if you have any questions. To find out if you may be affected by this maintenance, go to the DreamHost panel to check the location of your services. Comments will be turned on during the maintenance to assist you in real time, should we experience any unexpected difficulties.

Update Oct. 15th 12:45am PDT: This maintenance is now resolved. Our team monitored the changes to make sure services remained working. We have not seen any issues. We will have our monitoring staff keeping an eye on things, but this maintenance is now considered resolved and completed with 0 customer impact. Thank you for your patience!

Previous updates available after the jump →


11:09 pm

Network maintenance – upstream provider and data center – US-West Data Center (Los Angeles) Thursday Oct 10th 10pm PDT (RESOLVED)

Posted (October 9th, 2013 at 11:09 pm PST) by Ralph

On Thursday October 10th starting at 10pm PDT, there will be a maintenance at our US-West (Los Angeles) data center. This maintenance will be conducted by one of our upstream providers and the data center staff.

To ensure this goes smoothly we will also have our own team on site to help assist the providers in any way possible, or to respond to any emergency situations that may come up.

This maintenance involves moving network links from one physical location to another. The links that are being moved are redundant, and will be moved one by one. Even though he services from this upstream provider will be affected by this maintenance, we don’t expect this to have any impact on customer services. Regardless, we will stand by, so that we can communicate if anything comes up. Comments will be opened once the maintenance begins, so that we can assist you with any possible related issues and concerns.

Please contact support if you have any questions. To find out if you may be affected by this maintenance, go to the DreamHost panel to check the location of your services.

Update Oct. 10th 11:00pm PDT: This maintenance is now resolved. There was 0 impact to the services we offer. If you are noticing any issues, please contact our support team.

Previous updates available after the jump →


7:03 am

Webmail Connection Errors ( RESOLVED )

Posted (October 9th, 2013 at 7:03 am PST) by Dan J

We’re currently investigating errors being reported by the webmail.dreamhost.com servers. This is impacting webmail connecting to all email servers. Direct IMAP/POP/SMTP connections from an email client such as Thunderbird or Outlook also appear to be experiencing intermittent connections issues. We should have more info shortly.

Update: October 9, 2013, 11:03 am PDT: We are happy to report that the mail service has been stabilized. You should no longer encounter any issues when accessing your email. We sincerely apologize for any inconvenience this has caused. If you are still seeing any kind of mail related issues then please contact support.

Previous updates available after the jump →


1:11 pm

Load Balancer Maintenance US-West Data Center (Los Angeles) – Redundant Power Upgrade – October 1st 8pm PDT (4 hours) (RESOLVED)

Posted (September 27th, 2013 at 1:11 pm PST) by Brian@dreamhost

On October 1st at 8pm PDT, we will be conducting maintenance on our Load Balancers in order to increase the power redundancy. The maintenance is scheduled to last 4 hours. We plan to move the hardware over in a series of monitored and pre-tested steps which should eliminate most service impact. Services that could be affected by this maintenance are listed below:

*All Mail Services – webmail – POP/IMAP – SMTP
DreamHost Web Panel

Our support team will be monitoring this status post and opening it up for comments once the maintenance starts. If you have any questions or concerns until then, please contact support.

*- Please note, even though the maintenance is in our US-West Data Center (Los Angeles), all customer Mail service and Panel service can possibly be affected. Mail sent from hosting machines, such as formmail responses, for example, will not be affected by this maintenance.

Update: October 1, 2013, 11:04 pm PDT: All done! Everything went according to plan and we’re resolving this maintenance! We thank you for your patience! If you are still seeing issues, or have any questions regarding this, please contact support and let us know so we can help!

Update: October 1, 2013, 10:23 pm PDT: Software update is now complete, we’re moving on to changing some network locations around. This can possible affect mail, specifically delivery, as well as checking and sending mail. We’ll update this as we move along.

Previous updates available after the jump →


11:21 pm

Live Chat Unavailable Monday September 23rd 10am – 1am PDT

Posted (September 20th, 2013 at 11:21 pm PST) by Brian@dreamhost

Our Live Chat support system will be unavailable this coming Monday, September 23rd, from 10am to 1am PDT (the following morning). We will be involving our support staff in a special, employee appreciation day, rewarding them for their excellent service and hard work! We aim to offer the best service possible and we feel our team deserves it!

Email support operations will still be available, should you need to get a hold of us. We will resume normal Live Chat operations at 1am PDT the next morning, September 24th. We thank you for your patience and understanding.


12:35 am

Server configuration issue impacting a small subset of VPS servers

Posted (September 17th, 2013 at 12:35 am PST) by mir

We are currently investigating an issue with one or our network configuration scripts which is disrupting networking for VPS guest servers on a number of VPS host machines. No other services (shared hosting/email/database/panel/etc.) are affected and we are in the process of fixing all host machines and their guests while also looking into the root case so we can also correct that. Currently under one tenth of customer VPS guests (and the sites on them) have been disrupted. We apologize for the inconvenience and will provide updates as they become available.

Update: September 17, 2013, 2:55 AM PDT: The clean up is finished and everything looks good. We are calling this resolved. Please contact us (Support -> Contact Support in the admin panel) if you are still experiencing any issues on your VPS. Thank you for your patience!

Previous updates available after the jump →


3:04 pm

WordPress 3.6.1 Security and Maintenance Release (RESOLVED)

Posted (September 11th, 2013 at 3:04 pm PST) by Mika E.

WordPress 3.6.1 is now available for new installs and updates from the DreamHost One-Click Install panel. Please be aware that this is a security release, and as such, it is highly recommended for all customers.

You can find the release notes on WordPress.org, and further details in the codex.

Unless disabled, automatic upgrades for DreamPress and other One-Click installs will begin shortly (of course, you would never disable them, right?). If you would like to see if you have automatic upgrades enabled, or want to run your own, you can do so directly from the DreamHost Panel.


 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
1:57 pm

Emergency email hardware maintenance on homiemail-sub4 cluster – RESOLVED

Posted (September 11th, 2013 at 1:57 pm PST) by andrea

Due to the issues we had with our email service yesterday and today on our homiemail-sub4 cluster, we feel it’s in our customers’ best interest to go ahead with a network maintenance. This is to alleviate the problems that have been popping up yesterday and today. Our System Engineering and Network Engineering teams have staged and tested the process of swapping out the hardware to a more robust switching infrastructure, and we aren’t expecting any customer-facing downtime. The process itself will take less than a minute. We are posting this to let you know that we’re working toward further stabilizing our email services. We appreciate your patience and apologize for all inconvenience this has been causing.

This maintenance is related only to email services on said cluster. To check if you’re affected, please go to your DreamHost panel. If you have any questions or concerns about this or anything else, please contact support with the details.

Update: September 11, 2013, 2:49 PM PDT: The maintenance was completed successfully. If you are still seeing issues, or have any questions regarding this, please contact support and let us know so we can help!


7:56 pm

Emergency Network Maintenance US-West Data Center (Los Angeles) Friday September 6th 9pm PDT (6 hours)(RESOLVED)

Posted (September 6th, 2013 at 7:56 pm PST) by Brian@dreamhost

Due to the network problems outlined here we are having to perform emergency maintenance tonight at 9pm PDT. This maintenance should last approximately six hours

This procedure will move more networks over to our new hardware, very similar to what we did here last thursday night, however, because of the emergency nature of this procedure, there are several larger differences.

-First, we anticipate some mail service interruption. This includes mail delivery, IMAP, POP and SMTP as well as webmail and mailboxes.

-Second, this will also affect core services like https://panel.dreamhost.com.

We apologize for the very short notice, and possible disruptive nature of this maintenance, however, we feel it’s absolutely needed in order to stabilize the network service we provide our customers. As always, we will be providing details the maintenance progresses

Update September 7th 1:50PM PDT: We have cleaned up any remaining issues and we are resolving this maintenance.

Previous updates available after the jump →


3:04 pm

Connectivity issues at US-West (Los Angeles) Data Center (RESOLVED)

Posted (September 6th, 2013 at 3:04 pm PST) by Sean

At this time, we are currently experiencing connectivity issues at our US-West (Los Angeles) Data center. Our teams are investigating the root cause and are working towards a resolution. We’re very sorry for the inconvenience this is causing, and will post updates as the work progresses. Web panel access, webmail and mailboxes (in addition to web, mysql and email) may be affected by these problems.

To see if you may be affected, please check your location in the DreamHost panel.

Please follow here for updates, and thank you for your patience!

Update August 6th 8:50PM PDT: We’re resolving this status post for now, and rolling these problems into the maintenance coming up tonight at 9pm PDT on friday, september 6th 2013. You can view that post here

Previous updates available after the jump →


11:34 pm

Google Authenticator Update for iOS Loses Account Information

Posted (September 3rd, 2013 at 11:34 pm PST) by Steven

Word on the internet is that the latest update to the iOS Google Authenticator app may lose account information on upgrade. If you have it set up on your DH account, this will lock you out of the admin panel (https://panel.dreamhost.com)! This is, obviously, bad. We recommend against upgrading the app (on iOS only) for the moment.


9:31 am

Network Maintenance US-West Data Center (Los Angeles) Thursday September 5th 8pm PDT (6 hours) (RESOLVED)

Posted (September 3rd, 2013 at 9:31 am PST) by Ralph

On Thursday, September 5th at 8pm PDT, we will be replacing our core network hardware in our Los Angeles US-West data center.

The maintenance window is being set at 6 hours to allow us to move methodically so we can limit any potential service degradation. The process will be complete in a series of steps.
Our initial step will require a reboot of our border hardware. During this step there will be degraded service for a period that is expect to last under 30 minutes, however sites will be up and working during that time.

Once that is confirmed complete our new hardware will be connected to the border and we will carefully move subsets of our network from the existing core, to the new core. In our previous maintenance we have been able to accomplish this with minimal to no impact on site availability, barring any unforeseen issues. If we do come across a problem we can easily roll the change back to previous core.

When all virtual networks are moved to the new core we will have completed all major network hardware replacements for the US-West Data Center (Los Angeles). Comments will be open and monitored during this maintenance so that we can work with you and assist with any issues that may come up during the maintenance.

As always, please contact support if you have any questions. You can also check if you may be affected by this in your DreamHost Web Panel by checking to see if you have services in the US-West data center (Los Angeles).

Update – September 6th, 12:20AM PDT: We’re done! Everything went according to plan and we’re resolving this maintenance just shy of 2 hours early! If you’re seeing problems with your site, please contact support asap!

Previous updates available after the jump →


3:55 am

Mail connectivity issues affecting homiemail-sub3, homiemail-sub4, homiemail-master and webmail services (RESOLVED)

Posted (August 30th, 2013 at 3:55 am PST) by chalise

We are currently investigating stability issues on the machines that handle mail service on three of our mail clusters, as well as webmail which is causing connection errors when trying to send or receive.

You can confirm whether you are affected by visiting our Web Panel https://panel.dreamhost.com and clicking the “Account Status” tab. Scroll to the bottom and you will see a field labeled “Your Email Cluster”.

We apologize for the inconvenience and greatly appreciate your patience while our team works to resolve this.

Update – August 30th, 4:50AM PDT: This is now resolved and all mail servers and webmail services are working normally once again. If you’re noticing any remaining mail issues, they are not related, so please be sure to contact support under “Support” / “Contact Support” in your panel. Thanks again for your patience!

Previous updates available after the jump →


4:38 pm

Live Chat Unavailable Wednesday August 28th 9am – 10pm PDT

Posted (August 26th, 2013 at 4:38 pm PST) by Ralph

Our live chat support service will be unavailable this coming Wednesday August 28th from 9am to 10pm PDT. We will be having a company meeting which will require our live chat team to be present. Email support operations will still be available. We will resume normal Live Chat operations at 10pm PDT if not earlier. We thank you for your patience. We will aim to get live chat up and running as soon as we can once the company meeting is complete.


8:32 am

One Click Installs/Upgrades Temporarily Offline (RESOLVED)

Posted (August 6th, 2013 at 8:32 am PST) by Oscar

Due to a hardware issue with one of the servers which processes the installations/updates of our one click installs, we’ve had to temporarily disable the installation and upgrades for all OCI’s. This will not affect your OCI if it was previously installed, it’s only affecting any new or pending upgrade OCI’s. We are working to restore all OCI functionality and we will update this post as soon as we have more information.

Update August 6th 2:10PM PDT: Great news! All one click installs and upgrades are now being processed again. There will be a delay before your one click install and upgrades are processed, but you will receive an email from us when complete. We sincerely apologize for any inconvenience that this delay has caused you. Should you have any issues with your OCI’s, please contact our support team and we will be glad to look into the issue for you.

Previous updates available after the jump →


2:12 pm

Connectivity issues at US-West and US-East Data Centers (RESOLVED)

Posted (August 3rd, 2013 at 2:12 pm PST) by Elizabeth

We are seeing connectivity issues at US-West Data Centers (Los Angeles and Irvine) and our US-East Data Center. Our teams are investigating the root cause and are working towards resolution. We are very sorry for the inconvenience this is causing, and will post update as the work progresses. To see if you may be affected, please check your location in the DreamHost panel.

We’re sorry for any issues and thank you for your patience!

Update August 3rd 2:45PM PDT: And this is resolved! You should not be seeing any problems with connection at this point. If you are having any issues with your sites, please contact support, so we can assist you.


4:00 pm

WordPress 3.6 Major Release Now Available (RESOLVED)

Posted (August 1st, 2013 at 4:00 pm PST) by Mika E.

Rabbit, Rabbit!

The newest version of WordPress 3.6 (Oscar), is now available on the DreamHost panel for new installs and upgrades. You can also upgrade from directly within your site’s WordPress admin panel, if you prefer.

3.6 is both a feature and maintenance release, and is highly recommended for all customers. Its new features include built in playback for audio and video, a bold new default theme, improved autosave and more!

For more details, you can check out the release notes here, with further details available in the WordPress codex.

If you run into issues after your upgrade, as a first step, take a look at this thread on 3.6 in the DreamHost forums, and the 3.6 master list at the WordPress.org forums, where issues relating to upgrades are being tracked and updated as they come up.

Over the next few days, this update will be automatically handled for all customers who have automatic upgrades enabled, including all those on DreamPress. If you used our One-Click Installer to install WordPress, you can check your settings, and change these preferences directly, by visiting the DreamHost One-Click Panel. Once there, click “Manage Installed Applications,” select the domain, then look under the Upgrade Action column for details.

You may want to take this time to also clean up your site’s themes and plugins. Delete anything you’re not using, and upgrade everything else. And don’t forget to make a backup!

You should really go watch the video. It’s pretty cool.


 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
5:30 pm

Emergency Hardware Maintenance US-East (Virginia) Data Center, Thursday Aug 1st 8pm PDT (4hrs) (RESOLVED)

Posted (July 31st, 2013 at 5:30 pm PST) by andrea

We’re scheduling a vendor-suggested emergency maintenance for Thursday evening, 8pm PDT. The maintenance itself will be rolling reboots of redundantly set up hardware. Because of this set up, we don’t estimate any customer-facing downtime, but our teams will be closely monitoring the process, and we also have a high-level support technician standing by on the vendor’s end too. We estimate the process to take no longer than two hours, but we’re setting a four-hour window to run all necessary tests and confirmations.

We apologize for the late notice, but we feel that this will greatly improve performance without affecting any services in the process. If you have any questions, please don’t hesitate to contact support. We’ll update this status post as soon as the maintenance starts and will be posting updates throughout.

Update August 1st 8:02pm PDT: We are ready to proceed with the maintenance. We’ll be posting updates as we progress.

Update August 1st 8:31pm PDT: So far so good; the maintenance is going smoothly. We’ll report back again as we go through the stages.

Update August 1st 9:11pm PDT: We are happy to report that we finished way ahead of schedule and without any impact on our services. The maintenance is now successfully completed! If you are having any sort of issues with any of your hosting services, it’s unrelated to this maintenance, so please contact support with the details, and we’ll be happy to assist.


3:45 pm

Secure certificate changes coming for SMTP on all mail clusters on June 27th 2:30pm PST (RESOLVED)

Posted (June 26th, 2013 at 3:45 pm PST) by andrea

We are happy to report that we are in the final stages of changing the secure certificates for all mail services for all our email clusters. A while ago, we rolled out the changes to IMAP and POP, and tomorrow, on the 27th, we’ll be doing the same for SMTP.

If you’re not using secure connection to check mail, you may ignore this announcement. If you do, however, here is a quick notice for you to prepare: you will likely run into a warning when you log into your email client again (after the change!), stating that the certificate has changed. No need to worry, as this was done by us. You also don’t have to do anything other than accepting the secure certificate if your client has an option for that. In some clients, you may have to drop the old certificate and restart the client then accept the new one.

Unfortunately, we can’t anticipate all email client behavior, but we would be happy to help, so please contact support if you need help sorting this out after we made the change.

Update July 2nd 11:11am PDT: The changes have been applied last week, and we’ve been monitoring them to make sure everything went well. Please contact support if you’re running into any issues with your mail SSL certificate, and we’ll be happy to help.


1:39 pm

WordPress 3.5.2 Security and Maintenance Release (RESOLVED)

Posted (June 21st, 2013 at 1:39 pm PST) by Mike S

WordPress 3.5.2 is now available for new installs and updates from the DreamHost One-Click Install panel. Please be aware that this is a security release, and as such, it is highly recommended for all customers.

You can find the release notes on WordPress.org, and further details in the codex.

Unless disabled, automatic upgrades for DreamPress and other One-Click installs will begin shortly (of course, you would never disable them, right?). If you would like to see if you have automatic upgrades enabled, or want to run your own, you can do so directly from the DreamHost Panel.

Update July 3rd 12:56pm PDT: This is finished as of Monday, July 1st, afternoon. If your install did not upgrade, or you run into further issues, please contact Tech Support via the DreamHost panel for assistance!

Previous updates available after the jump →


4:46 pm

Network Maintenance US-East Data Center (Virginia) Data Center Friday June 21st 10pm PDT (4 hours) (RESOLVED)

Posted (June 19th, 2013 at 4:46 pm PST) by Brian@dreamhost

We are conducting network maintenance in our US East Data Center (Virginia) on Friday, June 21st at 8pm PDT. This maintenance involves moving networking routes to brand new equipment, while directing traffic through thoroughly pre-tested and staged backups.

The maintenance is set for a four hour window, and we estimate most of this to not be customer-facing.
To see if you may be affected, please check your data center location for mail, web and mysql services in your DreamHost panel.

We will post updates as the maintenance progress and will be monitoring comments. If you have any questions or concerns, please contact support with the details, so we can assist you.

We appreciate your continued patience while we improve our networking services.

Update June 22nd 01:13am PDT: We’re all done and are resolving this status post as we have successfully completed the maintenance. Thank you for your patience throughout this maintenance!

Previous updates available after the jump →


12:33 pm

Network Maintenance US-West Data Center (Los Angeles, CA) Wednesday June 19th 10pm PDT (4 hours) (RESOLVED)

Posted (June 19th, 2013 at 12:33 pm PST) by Brian@dreamhost

Tonight, Wednesday June 19th, we will be performing maintenance on the network at our US-West Data Center (Los Angeles, CA). We will be starting at 10pm PDT with a 4 hour window. During the maintenance we will be restarting the links between our border and core routers, as well as our link connecting our data centers. The redundant network should give us the ability to perform these changes without impacting your service. Our teams will be on hand if anything unexpected should occur during this window. We will also open up comments and monitor them throughout the maintenance to ensure any concerns are addressed immediately.

Update June 19th 11:06pm: Our network remains stable, and the maintenance is now complete. We appreciate your continued patience!

Previous updates available after the jump →


10:23 am

Mail connectivity issues affecting homiemail-sub3 and homiemail-sub4 (RESOLVED)

Posted (June 18th, 2013 at 10:23 am PST) by Oscar

We are currently investigating stability issues on the machines that handle mail service on two of our mail clusters. This accounts for the connection errors which customers on sub3/sub4 are experiencing at the moment. This is also causing webmail to timeout. You can confirm whether you are affected by visiting our Web Panel https://panel.dreamhost.com and clicking the ” Account Status ” tab. Scroll to the bottom and you will see a field labeled ” Your Email Cluster “.

We apologize for any inconvenience this has caused. Our team is diligently working towards restoring service and we will continue to post updates as information becomes available.

Update June 18th 11:40am: We have received the “all clear” from all of our teams. All mail services are up and responding without any issues for homiemail-sub3 and sub4. If you are having any issues with your email now, please contact our support team as it is no longer related to this outage. You can contact our support from your web panel or click here. Thank you for your patience while we resolved this issue.

Previous updates available after the jump →


6:41 am

Networking issues affecting both US-West Data Centers (Los Angeles and Irvine) (RESOLVED)

Posted (June 13th, 2013 at 6:41 am PST) by Oscar

Emergency network maintenance: Due to the failed optical card on one of our core routers that affected our US  West Los Angeles and Irvine data centers earlier this morning, our network engineering team is going to perform an emergency maintenance to replace the failed component. This will help to solidify our network from this type of incident happening again.

We are estimating two hours for this maintenance, to adequately pre-test and make changes and test again. We don’t expect this procedure to cause any noticeable service interruptions. Once we have started, we will be monitoring your status post comments to make sure everything is going as planned. We will update this status post as soon as the maintenance commences and will be updating it with progress reports.

Update June 13th 10:15am PDT: The maintenance is now complete and we have received the “all clear” from all of our teams. All services are up and responding without any issues. If your sites are down or not resolving, please contact our support team as it is no longer related to this maintenance. You can contact our support from your web panel or click here. Thank you for your patience throughout the initial outage as well as the emergency maintenance.  Previous updates available after the jump →


3:13 pm

Package Upgrade on Dedicated Servers – Friday June 14th 11am PDT (RESOLVED)

Posted (June 12th, 2013 at 3:13 pm PST) by Oscar

This Friday we will upgrading the fcgi package on all dedicated servers. We will be starting at 11am PDT on June 14th   and anticipate this upgrade completing on June 15th. We do not anticipate any downtime for any of our dedicated customers during this upgrade.  Our teams will be on hand if anything unexpected should occur during this window. We will also open up comments and monitor them throughout the maintenance to ensure any concerns are addressed immediately.

Update June 14th 2:15pm PDT: The package upgrade was completed earlier this morning and after final checks, we are now closing this status post. The upgrade was completed without any issues, but required apache instances to be rebooted. If you have any issues with your dedicated servers or sites, please contact our support from your web panel or click here.


5:31 pm

Network Maintenance US-West (Irvine) Wednesday June 12th 10pm PDT (3 hours)(RESOLVED)

Posted (June 10th, 2013 at 5:31 pm PST) by Ralph

This Wednesday we will be performing maintenance on the network at our US-West (Irvine) data center. We will be starting at 10pm PDT on June 12th with a 3 hour window. During the maintenance we will be modifying the links between our border and core routers, as well as our link connecting our data centers. The redundant network should give us the ability to perform these changes without impacting your service. Our teams will be on hand if anything unexpected should occur during this window. We will also open up comments and monitor them throughout the maintenance to ensure any concerns are addressed immediately.

To see if you’re affected by this maintenance, please check your location in the DreamHost panel.

Update June 13th 12:15am PDT: Okay we are resolving this maintenance. We have successfully completed the maintenance. Thank you for your patience throughout this update!

Previous updates available after the jump →


2:42 pm

Outgoing mail delays, slowness, and disconnection errors in ‘homiemail-master’ and ‘homiemail-sub4′

Posted (June 4th, 2013 at 2:42 pm PST) by Seohee

Our system engineers are noticing elevated loads on some machines, which is causing a delay in the processing of sent mail as well as slowness or intermittent disconnections in your email client and webmail. Our administrators are working diligently on getting the loads down, as well as working on a longterm solution to improve the stability and service as a whole. Unfortunately, there can be intermittent reoccurrences until this is complete.

You can log into our Web Panel and confirm whether you are impacted by this outage. Click the “Account Status” link. Again, you are only impacted if “homiemial-master” or “homiemail-sub4″ is listed in the field labeled “Your E-mail Cluster”.

We will provide you with progress updates on the status of the longterm solutions on this post. We sincerely appreciate your patience and are sorry for the trouble this has caused.

Update June 15th, 4:56 PM PDT: The new setup and the new machines are making serious improvements on the delays. We are still considering solutions for the longer-term, but for now you should not be seeing issues — if you are, please let us know here. Thank you for your patience!

Previous updates available after the jump →


 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
3:50 am

DreamObjects service disruption (RESOLVED)

Posted (May 30th, 2013 at 3:50 am PST) by chalise

We are currently experiencing problems with our DreamObjects service. Our team is looking into this, and we will provide further updates as more information becomes available.

Thanks for your patience!

Update May 30 4:45am PDT: All services are now back online, and our team will continue to look into the cause, and how we can prevent any similar issues in the future. Thanks for your patience while we got everything back up and running!

Previous updates available after the jump →


6:42 pm

Alchemy (US-West Irvine Data Center) UPS power and battery preventive maintenance Tuesday, May 28 11pm – May 29th 7am Pacific Time (RESOLVED)

Posted (May 23rd, 2013 at 6:42 pm PST) by Ralph

Our data center provider, Alchemy, will be conducting preventative maintenance on their UPS power and batteries in the US-West (Irvine) Data Center. This maintenance will require the building UPS system to be transferred from the city to the generators during part of the window. Once complete the power will be transferred back to the city power again.

A UPS system is the battery backup that sustains the load of the data center when utility power is cut. Simply put, the job of the UPS system is to keep the lights on when transferring from utility power to generator power, and back again. Think of it as a battery backup for your computer on an extremely large scale.

Alchemy states:
“The maintenance should be transparent to you and no interruptions of service are anticipated; however as with any maintenance, there is a small possibility of service interruptions as changes are made. We expect the duration of the work to be approximately eight (8) hours.”

We will have our teams on hand to make sure, that should any interruption occur, we will be ready with a plan to immediately restore service. At this time we are confident that there will be no service interruption, however we plan to be prepared and have our teams responding to comments to this post and updating you with anything we hear during the maintenance.

To see if you have services in the US-West Irvine data center, please check your data center location in the DreamHost panel.

Update May 29th 2:42am PDT: Maintenance has now completed, without any service disruptions!

Previous updates available after the jump →


12:22 pm

Mail connectivity issues affecting homiemail-sub4 and homiemail-sub5 (RESOLVED)

Posted (May 22nd, 2013 at 12:22 pm PST) by Oscar

We are currently investigating stability issues on the machines that handle mail service on two of our mail clusters. This accounts for the connection errors which customers on sub4/sub5 are experiencing at the moment. This is also causing webmail to load slowly. You can confirm whether you are affected by visiting our Web Panel https://panel.dreamhost.com and clicking the ” Account Status ” tab. Scroll to the bottom and you will see a field labeled ” Your Email Cluster “.

We apologize for any inconvenience this has caused. Our team is diligently working towards restoring service and we will continue to post updates as information becomes available.

Update May 22nd 1:50 PM PDT: We are no longer seeing any intermittent connectivity issues. At this time you should be able to connect to your email clients and webmail without any issues. Our network engineering team is monitoring all mail machines to ensure this issue doesn’t pop up again and is investigating options to improve overall mail stability. We apologize for the disruption of service as we understand how important mail is.

Previous updates available after the jump →


4:26 pm

DreamObjects Maintenance – May 22nd through Friday May 31st (RESOLVED)

Posted (May 20th, 2013 at 4:26 pm PST) by mir

Our Systems Engineering team will be performing maintenance on our DreamObjects service from Wednesday, May 22nd through Friday, May 31st. During this maintenance period you may intermittently experience degraded performance. The purpose of this maintenance is to implement key upgrades to Ceph, the underlying storage system which powers DreamObjects. As a result of the upgrades our development team will be able to provide some exciting new features, including CNAME support, in the near future. While we anticipate slower than normal speeds when storing or retrieving objects, please rest assured that your data is safe and will not be affected.

If you have any questions or concerns, please don’t hesitate contact support for assistance.

Update May 31st 4:20 PM PDT:  We are happy to report that the maintenance is now complete. As mentioned before, this upgrade allows for new and exciting features, like CNAME support. We are working on implementing these and more right now, and will be sharing the news with you soon through the Cloud segment of our Dreamhost.com and blog posts, so keep an eye out for them!

Previous updates available after the jump →


2:24 pm

Mail connectivity issues on homie-sub3 and homie-sub4 ( RESOLVED )

Posted (May 20th, 2013 at 2:24 pm PST) by brians

Our Systems Engineering Team is currently investigating stability issues on the machines that handle mail service on two of our mail clusters. This accounts for connection errors which customers on the impacted mail clusters are experiencing at the moment. You can confirm whether you are affected by visiting our Web Panel https://panel.dreamhost.com and clicking the ” Account Status ” tab. Scroll to the bottom and you will see a field labeled ” Your Email Cluster “.

We apologize for any inconvenience this has caused. Our team is diligently working towards restoring service and we will continue to post updates as information becomes available.

UPDATE May 5th, 3:09pm PDT

The impacted mail service has been restored and you should now be able access the server again. Once more, we sincerely apologize for any inconvenience this has caused. If you are still experiencing service disruption please contact support!

Previous updates available after the jump →


10:01 am

Connectivity issues affecting our US-West Data Center (Los Angeles) (RESOLVED)

Posted (May 17th, 2013 at 10:01 am PST) by Oscar

We are currently experiencing some connectivity issues in our US-West Data Center (Los Angeles, CA). Our teams are investigating the root cause and are working towards resolution. We are very sorry for the inconvenience this is causing, and will post update as the work progresses. Any changes requested in the panel will be delayed throughout this outage. To see if you may be affected, please check your location in the DreamHost panel.

Update May 17th 11:25am PDT: All services are reporting back online now. Any new domain registrations attempted during this outage may have failed and will need to be re-attempted. We appreciate your patience while we resolved this outage. If you are having any issues with your sites, any changes made in the panel or domain registrations please  contact support, so we can assist you. Previous updates available after the jump →


4:58 pm

VPS web service maintenance May 15th-18th — Extended till May 24th (RESOLVED)

Posted (May 15th, 2013 at 4:58 pm PST) by andrea

We are conducting preventative maintenance on VPS web services starting 5pm PDT today. This is the first batch, and we’ll be going through all our VPS machines which will take us approximately three days, in segments.

Apologies on the late notice.  Since this fix was integral to stabilizing our servers after the brute force attacks earlier this week, we feel it’s necessary to roll out the changes immediately. This maintenance will require the web servers to be restarted after the change has been applied. The restart will cause minimal (few seconds) downtime per server.

We will keep updating this post with progress reports and monitor the status comments in case further action is needed on our end.

Update May 24rd 4:35 PM PDT: The upgrades are now complete! I am very sorry that it took longer than planned. If you run into any issues, please contact support so we can help you!

Previous updates available after the jump →


2:43 pm

Emergency software maintenance in our US West (Los Angeles) Data Center May 15th, 8pm (4 hrs) (RESOLVED)

Posted (May 15th, 2013 at 2:43 pm PST) by andrea

In light of what happened earlier today in our US West (Los Angeles) Data Center, we are going to perform an emergency maintenance tonight to help further solidify our network from this type of incident. We are still looking into the exact root cause of the incident – keep on a lookout for a detailed status post about it tomorrow.

During tonight’s emergency maintenance, we will be adding a patch to the core router software.

As our CEO mentioned in an earlier status post comment, we have been continuously upgrading both our hardware and software in all our data centers. We understand, and also experience, the frustration these continued situations are causing. We are working closely with the vendors as we progress through these network upgrades and changes.

We are estimating four hours for this maintenance, to adequately pre-test and make changes and test again. We don’t expect this procedure to cause any noticeable service interruptions. Once we have started, we will be monitoring your status post comments to make sure everything is going as planned. We will update this status post as soon as the maintenance commences and will be updating it with progress reports.
To see if you’re affected by this maintenance, please check your location in the DreamHost panel.

Update May 14th 11:20pm PDT: That went perfectly as well, so we’re closing this one for good!

Previous updates available after the jump →


10:02 am

Connectivity issues affecting subset of customers in US-West Data Center (Los Angeles, CA) (RESOLVED)

Posted (May 15th, 2013 at 10:02 am PST) by Justin@DH

Some of our customer in our US-West Data Center (Los Angeles, CA) may be having some connectivity issues at the moment. This includes web, mysql services, and specifically the homiemail-sub5 mail services (You can find this in the panel via Account Status on the top right, and checking “Your Email Cluster”).
To check if you’re in the affected data center, please check your location in the DreamHost panel.

We are investigating the root cause and are working towards resolving it. We are very sorry for the inconvenience this is causing, and will post update as soon as we have more information.

Update May 15th 12:45pm PDT: We believe the connectivity issues to be resolved at this time. There will need to be a network maintenance tonight, and a new post will be up soon with details. Again our apologies for the downtime caused by this and your patience as we complete network upgrades and hardening.

Previous updates available after the jump →


10:45 am

Renewed external attacks affecting customers across all datacenters (RESOLVED)

Posted (May 14th, 2013 at 10:45 am PST) by danwy

We are seeing a new wave of attacks, similar to the ones that were causing issues yesterday. They have restarted and are causing new problems again today, and despite targeting wordpress, affect all customer on our web hosting services. Our admin team is aware of this renewed series of attacks and are already working on trying to restore services to normal. Again, we apologize for the inconvenience this causes you and we will post updates as soon as more information becomes available.

Update May 14th 4:33pm PDT:The update that our admin team pushed out has helped overall stability across our network considerably. As of now, the fix is running on all shared hosts and we are seeing a very positive result. We’ll continue to monitor the health of our machines, but the night and day difference hints that our team has very likely correctly identified and fixed the root cause of the issue. The attacks against WordPress haven’t particularly quelled, but they are not having nearly the negative effect on the machines they were prior to our latest patch rollout.

Previous updates available after the jump →


 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
11:19 am

External attacks affecting customers across all datacenters (RESOLVED)

Posted (May 13th, 2013 at 11:19 am PST) by Dan J

Some of our customer may be having some connectivity issues at the moment. This includes web and mysql services, but no mail services are affected. We are investigating the root cause and are working towards resolving it. Please accept our apologies for the inconvenience this is causing and rest assured that we will post updates whenever more information becomes available.

Update May 14th 10:38am PDT: We are seeing a new wave of these attacks that are causing issues. Please keep an eye on this new status post for updates.

Previous updates available after the jump →


9:13 am

Networking issues affecting US-West Data Center (Irvine, CA) (RESOLVED)

Posted (May 12th, 2013 at 9:13 am PST) by danwy

Some of our customer in our US-West Data Center (Irvine, CA) may be having some connectivity issues at the moment. This includes web and mysql services, but no mail services are affected. We are investigating the root cause and are working towards resolving it. We are very sorry for the inconvenience this is causing, and will post update as soon as we have more info.

To see if you may be affected, please check your data center location in the DreamHost panel.

Update May 12th 3:44pm PDT: We now consider this incident resolved; we’ve been monitoring connectivity for the past few hours and have not observed any further issues. If you’re still having problems with your web and mysql services, it’s unrelated, so please contact support with the details, so we can assist you.

Update May 12th 1:35pm PDTWe’ve been keeping an eye on things for the last hour and things have been running smoothly with no packet loss. We believe this issue is resolved at this time, but will keep this post open for a bit longer while we continue to keep an eye on things. If you continue to have issues, please contact support to get your problem looked into.

Previous updates available after the jump →


8:02 pm

Network maintenance in US West Data Center (Los Angeles), May 14th 6pm (8hrs) (RESOLVED)

Posted (May 11th, 2013 at 8:02 pm PST) by andrea

We are conducting network maintenance in our US West Data Center (Los Angeles) on Tuesday, May 14th at 6pm PDT. This maintenance involves moving networking routes to brand new equipment, while directing traffic through thoroughly pre-tested and staged backups.

The maintenance is set for an eight-hour window, and we estimate most of this to be not customer-facing.
To see if you may be affected, please check your data center location for mail, web and mysql services in your DreamHost panel.

We will post updates as the maintenance progress and will be monitoring comments. If you have any questions or concerns, please contact support with the details, so we can assist you.
We appreciate your continued patience while we improve our networking services.

Update May 15th 1:40am PDT: The IPv6 tests didn’t work as expected, so we are moving that network back to its original location. For now, the maintenance window is done. We will move IPv6 at a later date. We appreciate everyone’s patience while we performed this maintenance.

Previous updates available after the jump →


4:26 am

Connectivity Issues for subset of customers in US-West Data Center (Los Angeles, CA) (RESOLVED)

Posted (May 11th, 2013 at 4:26 am PST) by andrea

Some of our customer in our US-West Data Center (Los Angeles, CA) may be experiencing connectivity issues. This includes web and mysql services, but no mail services are affected. Our teams are investigating the root cause and are working towards resolution. We are very sorry for the inconvenience this is causing, and will post update as the work progresses.

To see if you may be affected, please check your data center location in the DreamHost panel.

Update May 11th 5:10am PDT: We now consider this incident resolved; all connectivity issues have been been handled and haven’t experienced anything related during the monitoring period. If you’re still having problems with your web and mysql services, it’s unrelated, so please contact support with the details, so we can assist you.

Previous updates available after the jump →


4:37 pm

Web Service Disruption Affecting Subset of Customers at US-West (Los Angeles) and US-East Datacenters (RESOLVED)

Posted (May 10th, 2013 at 4:37 pm PST) by Elizabeth

We are investigating intermittent service interruption affecting a small number of web servers at the US-West Los Angeles and US-East Virginia data centers right now, but we do not have an estimated time to solution yet.

If you are unsure if you are affected, you can check which datacenter you utilize in the control panel here.

We will let you know as soon as we have more information what the problem is and what we are doing to alleviate it.

Update May 10th 6:45pm PDT: We are no longer seeing problems with memory-related outages on any of our shared hosts. We are still investigating the root cause, but at this time we are resolving this issue. If you are still experiencing issues with your site, it is not related to this issue, so please contact support via Support / Contact Support in your DreamHost panel so we can help!

Previous updates available after the jump →


3:20 pm

Secure certificate changes coming for IMAP and POP on homiemail-sub4 and homiemail-sub5 email clusters on May 14th (RESOLVED)

Posted (May 9th, 2013 at 3:20 pm PST) by andrea

We are happy to report that we are getting closer to changing the secure certificates for IMAP and POP on all our email clusters. As you saw, it’s already rolled out on homiemail-master and homiemail-sub3 and we’re working on the final steps to make it happen on homemail-sub4 and sub5. We’re estimating the rollout date to be next Tuesday, May 14th. We will update this post when the changes are imminent and if there is any changes in the rollout schedule.
To check your mail cluster, log into your DreamHost panel and click your Account Status on the upper right hand side.

If you’re not using secure connection to check mail, you may ignore this announcement. If you do, however, here is a quick notice for you to prepare: you will likely run into a warning when you log into your email client again (after the change!), stating that the certificate has changed. No need to worry, as this was done by us. You also don’t have to do anything other than accepting the secure certificate if your client has an option for that.

For now, this is only affecting IMAP and POP access. We’re still working on and testing the options for the SMTP secure certificate change, and we’ll post again when we get closer.
We will post again when we roll this change out to the other email clusters, and when SMTP is ready to go as well.

This is good news for those trying to use gmail as POP or IMAP to access their DreamHost email, as they will be able use that service without warnings and problems.

Please contact support if you have any questions.

Update May 16th 10:15am PDT: The changes have now been applied to all of our mail servers. If you have any issues connecting to your email account within your email client or sending/receiving please make sure you make an exception for our new certificate. Once you have done so, you should have no issues connecting or sending/receiving. In some cases you may need to delete our old certificate in your email client and accept our new one and you should be all set.  If you have any questions or concerns, please contact support, so we can assist you.

Update May 15th 3:49pm PDT: We are now rolling out the changes to the above secure certificates. Our support team is standing by would you have any questions or concerns.

Update May 14th 2:19pm PDT: Due to recent events, we’re rescheduling this email cluster update to May 15th. We apologize for this, however, we feel its best to delay to ensure we have the focus needed to make this change. If you have any questions, please let our support staff know!


8:09 am

Connectivity Issues with US-West Data Center (Los Angeles, CA) (RESOLVED)

Posted (May 9th, 2013 at 8:09 am PST) by Oscar

We are currently experiencing issues with some servers in our Los Angeles data center. At this time, we are investigating the cause and we will let you know as soon as we have more information what the problem is and what we are doing to alleviate it. We sincerely apologize to those customers who are being affected by this DDoS attack. If you are unsure if you are affected, you can check which datacenter you utilize in the control panel here.

Update May 9, 9:00am PST:  Our network engineers and system administration team have blocked the DDoS attacks we were seeing to our Los Angeles data center. We will continue to investigate the cause of the attacks so as to prevent similar attacks in the future. If you are still experiencing issues with your site, it is not related to this issue, so please contact support via Support / Contact Support in your DreamHost panel so we can help!

Update May 9, 8:50am PST:  Our network engineers and system administration team are still working on blocking the DDoS attacks. We are making progress and narrowing down which machines are specifically being targeted. We will update this post with further information as it becomes available.

Update May 9, 8:30am PST: There seems to be DDoS activitiy which is causing downtime for a small subset of our customers. Our network engineers and system administration team are working to block the attacks at this time. We apologize to those customers who are affected and will update this post with further information as it becomes available.


10:47 am

Emergency network maintenance in our US-West Data Center (Los Angeles, CA) (RESOLVED)

Posted (May 6th, 2013 at 10:47 am PST) by andrea

We are starting an emergency network maintenance shortly, around 11am PDT, in our US-West Data Center (Los Angeles, CA). This will affect mostly mysql servers, and a very small subset of web servers. No mail services will be affected. We are not expecting any customer-facing issues, but our teams are closely monitoring this maintenance nonetheless.

As always, please contact support if you have any questions. You can check if you may be affected by this in your DreamHost panel.
We will update this status post as the maintenance moves forward and monitor comments posted here.

Update May 6th 11:00am PDT: This maintenance is complete and we are monitoring to make sure that no customer services are affected.

Update May 6th 11:20am PDT: We have monitored our services and found no issues related to the maintenance. If you see any issues that you believe may be related, please contact support.


2:55 pm

Secure certificate changes for IMAP and POP on homiemail-master and homiemail-sub3 email clusters (RESOLVED)

Posted (May 2nd, 2013 at 2:55 pm PST) by andrea

We are happy to report that your SSL connection to IMAP and POP is now using a non self-signed secure certificate. If you’re not using secure connection to check mail, you may ignore this announcement. If you do, however, you will likely run into a warning when you log into your email client again, stating that the certificate has changed. No need to worry, as this was done by us. You also don’t have to do anything other than accepting the secure certificate if your client has an option for that.

For now, this is only affecting IMAP and POP access for customers on the homiemail-master and homiemail-sub3 email clusters. To check your mail cluster, log into your DreamHost panel and click your Account Status on the upper right hand side.
We will post again when we roll this change out to the other email clusters, and when SMTP is ready to go as well.

This is good news for those trying to use gmail as POP or IMAP to access their DreamHost email, as they can now use that service without warnings and problems.

Sorry about the scare, and please contact support if you have any questions.

Update May 16th 10:15am PDT: The changes have now been applied to all of our mail servers. If you have any issues connecting to your email account within your email client or sending/receiving please make sure you make an exception for our new certificate. Once you have done so, you should have no issues connecting or sending/receiving. In some cases you may need to delete our old certificate in your email client and accept our new one and you should be all set.  If you have any questions or concerns, please contact support, so we can assist you. Previous updates available after the jump →


12:56 pm

DreamHost panel connection issues (RESOLVED)

Posted (May 2nd, 2013 at 12:56 pm PST) by andrea

We are currently experiencing connection issues to the DreamHost panel. It’s being investigated, and we’re expecting it to be online soon. Please note, this is also affecting your mailbox manager at mailboxes.yourdomainnamehere.com . We’ll post updates as this progresses. We apologize for the inconvenience this is causing.

Update May 2nd 1:06pm PDT: We’re happy to report that we restored connectivity manually, and the DreamHost panel and your mailbox manager is now back online again. We appreciate your patience. If you have any remaining issues, they are likely unrelated, so please contact support with the details, so we can assists you.


 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
5:39 pm

Networking issues in our Los Angeles Datacenter (RESOLVED)

Posted (May 1st, 2013 at 5:39 pm PST) by Seohee

We’re experiencing a network connectivity issue in our Los Angeles Datacenter. This will affect your sites, MySQL, email, webmail, the DreamHost panel, and Dreamhost.com. We’re currently investigating the cause and working on returning all services. We’ll provide an update shortly. You can check if you’re affected by this outage by looking up your data center location on your DreamHost panel.

We apologize for the trouble and appreciate your patience.

Update May 1st, 11:35PM PDT: We are confident that we have identified and resolved the hardware that caused this issue to occur. Thank you for you patience and communication during this incident.

Previous updates available after the jump →


4:26 pm

Network Maintenance US-West Data Center (Los Angeles, CA) – Core Router Software Upgrade – Thursday May 2nd 6:00pm PDT (9 hours, EXTENDED, RESOLVED)

Posted (May 1st, 2013 at 4:26 pm PST) by Ralph

On Thursday, May 2nd at 6pm PDT, we will be performing a core router software upgrade in our US-West Data Center (Los Angeles, CA).
We are scheduling a 6-hour maintenance window to allow a safe pace to test each step and monitor the successful completion. During this upgrade, the network traffic will temporarily flow through routers that have been thoroughly pre-tested, to minimize downtime. This is a routine maintenance that we have been able to accomplish in the past with seconds of impact on service.
As always, please contact support if you have any questions. You can check if you may be affected by this in your DreamHost panel.
We will update this status post as the maintenance moves forward and monitor comments posted here.

Update Thursday, May 3nd, 3:36am PDT:As of now, the maintenance window is now over, and the move to new hardware is complete! We thank you for your patience, and we apologize for having to extend beyond our original estimated completion time!

Previous updates available after the jump →


12:18 pm

Ongoing mysql hardware maintenance in Virginia (US-East) Data Center

Posted (May 1st, 2013 at 12:18 pm PST) by andrea

We are conducting mysql hardware maintenance in our Virginia (US-East) Data Center. This involves rebooting the servers, one at a time. This means about a 30 second downtime, and our data center operations team is closely monitoring the reboots to avoid any issues.

You can check if you may be affected by this in your DreamHost panel. We appreciate your patience while this maintenance is completed and apologize for all inconvenience caused. If you have any questions or concerns, please contact support, so we can assist you.

Update May 23rd 12:00 PM PDT: We have completed the needed changes and reboots for this maintenance. The servers affected will now have better connectivity and will be better able to be managed and monitored. Thank you for your patience.

Previous updates available after the jump →


4:33 pm

Emergency hardware maintenance in Virginia (US-East) Data Center – Tuesday, April 30th 8pm PDT (RESOLVED)

Posted (April 30th, 2013 at 4:33 pm PST) by andrea

Please accept our apologies for the late notice – we’re conducting emergency maintenance on the DOS attack mitigation hardware located in our Virginia (US-East) Data Center tonight, starting at 8pm PDT.
This is necessary due to the issues we experienced earlier today. We have been working with the vendor to diagnose the root cause, and will continue doing so later (at a less disruptive time of day for the majority of our users).

We are expecting this maintenance to be non-customer facing, but we are standing by to cover all possible situations. We appreciate your patience while we’re working on resolving this and apologize for any inconvenience this may be causing.

UPDATE: 6:28 PM PDT: We are resolving this issue as the DDoS mitigation hardware has been reported by our networking team as functioning correctly.  Please contact us if you still have any questions or concerns.
Previous updates available after the jump →


3:37 pm

Networking issues at US-East Data Center – RESOLVED

Posted (April 30th, 2013 at 3:37 pm PST) by Elizabeth

Our admins are working on an issue with our East Coast datacenter. According to their reports, one machine is blocking access to a section of the datacenter.  At this time we don’t have an ETA, but we will update this post as soon as possible.

UPDATE April 30th, 4:10 PM PST: The issue was related to the earlier outage. The problems are now resolved, and you should no longer see any issues. If you are still seeing problems with your hosting, please contact support. Thanks you for your cooperation and patience while we worked on this issue.

Previous updates available after the jump →


9:12 am

Connectivity Issues with Virginia (US-East) Data Center (RESOLVED)

Posted (April 30th, 2013 at 9:12 am PST) by Justin@DH

We are currently seeing issues to servers in our Virginia data center. At this time, we are investigating the cause and we will let you know as soon as we have more information what the problem is and what we are doing to alleviate it. If you are unsure if you are affected, you can check which datacenter you utilize in the control panel here.

UPDATE April 30th, 10:50AM PST – The cause of the network issues has been identified and is now resolved. We are continuing our investigation into what triggered this and will plan on how to prevent it happening in the future. If you are still experiencing issues with your hosting, please contact support. Thanks you for your cooperation and patience while we worked on this issue.

Previous updates available after the jump →


8:26 pm

Dreamhost.com Down (RESOLVED)

Posted (April 25th, 2013 at 8:26 pm PST) by Seohee

Our main page http://dreamhost.com and webmail are currently down. We don’t have any details as of this time, but will make sure to provide you with another update shortly. We thank you for your patience.

Update Apr. 25th 9:11PM PST: We found that the root cause was due to a central database being rebooted. We are looking into why the reboot happened, and why it came back in a different state than its original to prevent this from arising again in the future. Again, we appreciate your patience while we got this worked out.

Previous updates available after the jump →


7:04 am

Shared email downtime on homiemail-sub5 (RESOLVED)

Posted (April 20th, 2013 at 7:04 am PST) by Oscar

We are currently experiencing a few issues with our homiemail-sub5 mail server which our admin team is looking into. The outage should not result in any lost mail, however, you will be unable to access any email hosted on this server until this is resolved. This only affects any users on our homiemail-sub5 mail cluster. To check if you are affected, click the “Account Status” link on the top right of your DreamHost panel. Scroll to the bottom, and under “Your Email Server”, if it shows homiemail-sub5, this will explain any mail problems you are noticing at present. Please note, this only affects email hosted on homiemail-sub5. This does not affect any web or MySQL servers. We sincerely apologize for the inconvenience and thank you for your patience! We will post further updates below as they become available.

Update Apr. 20th 8:00am PST: Our network engineering team has resolved the issue and all mail connectivity has been restored on homiemail-sub5. There was no email lost during this outage, but there will be a delay of one hour or less for all mail to be processed and delivered. Again, we sincerely apologize for any inconvenience that this caused and we appreciate your patience while we resolved this issue.

Update Apr. 20th 7:23am PST: Our network engineers are working to resolve this issue and will we continue to update this post as we have more information. We apologize for any inconvenience that this is causing you.


1:15 pm

DreamHost Control Panel and Webmail Down (RESOLVED)

Posted (April 17th, 2013 at 1:15 pm PST) by Justin@DH

The DreamHost panel (https://panel.dreamhost.com) and webmail are currently unavailable. Our administrators are working on finding the cause, so we’ll make sure to provide you with further updates shortly.

We are sorry for the trouble, and appreciate your patience at this time.

Update Apr. 17th 1:18pm PST: The issue appears to be networking related and has been resolved. Webmail and the panel are working normally again at this time, and we will continue to monitor the services for any further issues. We appreciate your patience while we got this worked out, and are sorry for the trouble this has caused.


7:32 pm

Connectivity issues with some servers in Virginia (US-East) data center (RESOLVED)

Posted (April 10th, 2013 at 7:32 pm PST) by Elizabeth

We are currently seeing issues to a limited number of servers in our Virginia data center. At this time, we are investigating the cause and we will let you know as soon as we have more information what the problem is and what we are doing to alleviate it.

Previous updates available after the jump →


 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
8:32 pm

Mail Delays on our Junk Mail System (RESOLVED)

Posted (April 9th, 2013 at 8:32 pm PST) by Brian@dreamhost

We are currently experiencing mail delivery delays of up to a few hours affecting users who have our junk mail filtering enabled. Users without junk mail enabled are not affected.

The problems first started occurring this morning as a service stopped functioning on one of our junk filters. Further mails were successfully routed to other load balanced machines in the cluster, but one of them failed and was temporarily removed until load could be brought down. This unfortunately allowed the mail queues to grow rather large, resulting in the delays.

Our admins have determined the exact cause of the issue and will be looking into ways of further spreading out our mail load, as well as eliminating such single points of failure. The queue continues to diminish in size, so the situation is improving overall and delays are becoming less prevalent.

Update April 10th 12pm PDT: Our mail servers caught up last night, but unfortunately, we had a bit of a hiccup again this morning. It was much shorter in duration, and the mail delivery should be almost all caught up again. We are looking into procedures to avoid this situation again. Very sorry about the inconvenience this has caused.


2:19 pm

Connection issues in a segment of our network (US-West-Los Angeles) (RESOLVED)

Posted (April 9th, 2013 at 2:19 pm PST) by brians

Our System Engineers are noticing network problems on a specific row of machines in our Los Angeles,CA Data Center Facility. The affected machines are pasted below. We are diligently working towards resolving this issue. We will continue to provide updates as information is made available. Sorry for any inconvenience this has caused.

hoffa
batman
angel
biggs
dalitz
wolfsbane
karpis
provenzano
giancana
hornets
fitzpatrick
haddock
riddler
teak
vindemiatrix

UPDATE April 9th, 4:40 PM PST

Good news! All the machines are in relative good health, except for batman and riddler. We are closing this status post and communicate directly with customers on those two servers via email to keep them apprised of the situation.

If you are having any other issues, they are not related, so please be sure to contact support in your panel under Support / Contact Support.

Previous updates available after the jump →


5:52 pm

Web panel connectivity disruption (RESOLVED)

Posted (April 6th, 2013 at 5:52 pm PST) by Elizabeth

We apologize for the issues you may be having with accessing your Customer Web Panel at this time. We are working to resolve an issue which is causing this loss of connection and we will be updating this post with additional information as soon as possible.

Please note, this should not be affecting any other services at this time (web, mysql, email, etc.) and is only causing issues when trying to access your panel.

Thanks for your patience!

Update: 6:50 PM PDT: After a brief outage, our panel services are up once again. Thanks again for your patience. Even though we were able to resolve this situation quickly, we are looking into what we can take away from this case to put even better checks and monitoring in place in the next couple of weeks.  If you are having any other issues, they are not related, so please be sure to contact support in your panel under Support / Contact Support.

Previous updates available after the jump →


6:47 am

Connectivity issues with Web Panel (RESOLVED)

Posted (April 5th, 2013 at 6:47 am PST) by chalise

We apologize for the issues you may be having with accessing your Customer Web Panel at this time. We are working to resolve an issue which is causing this loss of connection and we will be updating this post with additional information as soon as possible.

Please note, this should not be affecting any other services at this time (web, mysql, email, etc.) and is only causing issues when trying to access your panel.

Thanks for your patience!

Update 6:50am PDT: Thanks for waiting! One of the machines running the panel was having a few issues rebooting, which was causing the panel to not load. This has now been resolved, and the source of the issues located, and all panel services are back up once again. Thanks again for your patience. If you are having any other issues, they are not related, so please be sure to contact support in your panel under Support / Contact Support.


12:58 pm

Connection issues in US-West Data Center (Los Angeles, CA) (RESOLVED)

Posted (March 29th, 2013 at 12:58 pm PST) by Oscar

We are currently experiencing packet loss in our Los Angeles data center. This is due to a massive DDoS (the largest we’ve seen) that is being directed to our network. Our network engineering team is working alongside our providers to block the attack and we are making progress. We will update this post as soon as we have more information.

Update March 29th 1:40pm PDT: At this time we are no longer seeing any issues and have confirmed that the DDoS attack has been fully mitigated. We will continue to monitor our network for any future attacks that may happen, but we are now resolving this post as customer services are no longer being affected. Thank you for your patience while we resolved this issue. Previous updates available after the jump →


7:43 pm

EMERGENCY MAINTENANCE: Network Improvement Series Maintenance – US-West Data Center (Los Angeles, CA) – Part 2/2 of Final Step – Wednesday, March 27th (RESOLVED)

Posted (March 26th, 2013 at 7:43 pm PST) by andrea

After some delays and postponements, we are excited to announce that we are at the end of our Network Improvement Series! This series of improvements started back in May 2012 is finally nearing completion. We started this project when we realized our network stability was in danger as we had outgrown our existing hardware. We have invested heavily and painstakingly planned to make sure we can provide you with the most stable network possible. On Wednesday, March 27th, we will be completely switching over to our new network infrastructure in the final data center US-West (Los Angeles, CA). We apologize for the late notice, but in light of yesterday evening’s incident, we feel it’s necessary to go ahead as soon as possible. This maintenance will start shortly and the window is 4 hours. During this maintenance we will be moving public traffic to our newly installed network, putting all live traffic on our shiny new network hardware. Much like our US-West final improvement series maintenance in Irvine, we will be moving traffic in small batches which will minimize impact and allow us to move at a pace that will limit service interruption. Services impacted during the maintenance will include Web, E-Mail including Webmail, MySQL, the Web Panel and other services located in the US-West (Los Angeles) Data Center. You can check if you are affected by this maintenance by seeing where your services are hosted in the DreamHost Web Panel.

As we progress through the maintenance we will monitor comments here and provide you with immediate assistance.

Update March 27th 9:00pm PDT: We have remained stable since our 4:10pm update and we are closing out this emergency maintenance. We will need to do a final maintenance which will be announced once we are ready to proceed with the final steps. As of now, the majority of our data center is on the upgraded hardware. Please be on the lookout for further maintenance announcements and thank you again for your continued patience.

Previous updates available after the jump →


5:35 pm

Connection issues in US-West Data Center (Los Angeles, CA) (RESOLVED)

Posted (March 26th, 2013 at 5:35 pm PST) by andrea

We are currently having connection issues in our US-West Data Center (Los Angeles, CA). We don’t have details yet, but our teams are already working on resolution. We’ll update this post as soon as we have more information.

Update March 27th 11:37am PDT: The problems appeared to be resolved then resurfaced, so with our CEOs approval, we have decided to go ahead with the emergency maintenance that was scheduled for tonight. Please see the status post above for details. We apologize for all the inconvenience this is causing.

Previous updates available after the jump →


3:01 pm

Power disruption affecting US-West Data Center (Irvine, CA)(RESOLVED)

Posted (March 19th, 2013 at 3:01 pm PST) by andrea

Emergency maintenance for normal power operations transfer. March 20th 9pm to March 21st 1am.

As of now the transfer back to UPS power and normal Irvine DC power operations is scheduled to start at 10pm. We are opening a 4 hour window for this emergency maintenance. We are pleased with the results of the fixes implemented and the testing that has thus far coincided. We do not anticipate this maintenance to cause another power incident. We will update this status post as we progress through this emergency maintenance.

Update March 20th 11:00pm PDT: Our data center providers emergency maintenance is complete and we are back on city power. This means we are operating under normal conditions with all redundancies back in place. We appreciate your time and patience throughout the issues we’ve experienced over the past day and a half. As always, if you have any questions or concerns, please contact tech support.

Previous updates available after the jump →


11:20 am

Emergency maintenance on DreamHost.com, FilesForever, Whois, DHurl, DreamHost wiki, and Discussion Forums March 19th 12pm PST (RESOLVED)

Posted (March 19th, 2013 at 11:20 am PST) by Ralph

At noon today we will be conducting hardware maintenance for the server responsible for the front end of DreamHost.com, FilesForever, our whois proxy, DHurl, Discussion Forums, and our customer wiki. This maintenance is to prevent possible unplanned problems with the server today. The maintenance will start at approximately 12pm PST and should last no longer than 1 hour. Our apologies for those using these site tools and the FilesForever service. We will update this post periodically as we progress through this short maintenance.

Update March 19th 12:05pm: This maintenance has now started. The services listed above will be out for a short period of time. If we encounter any complications we will update this post with further information.

Update March 19th 12:40pm: This maintenance is now complete as of 12:28pm. Thank you for your patience.


5:31 pm

POSTPONED: Network Improvement Series Maintenance – US-West Data Center (Los Angeles, CA) – Part 2/2 of Final Step – Wednesday, March 20th 8pm PST (4 hrs)

Posted (March 18th, 2013 at 5:31 pm PST) by andrea

We are excited to announce that we are at the end of our Network Improvement Series! This series of improvements started back in May 2012 is finally nearing completion. We started this project when we realized our network stability was in danger as we had outgrown our existing hardware. We have invested heavily and painstakingly planned to make sure we can provide you with the most stable network possible. On Wednesday, March 20th, we will be completely switching over to our new network infrastructure in the final data center US-West (Los Angeles, CA). This maintenance will start at approximately 8pm and the window is 4 hours. During this maintenance we will be moving public traffic to our newly installed network, putting all live traffic on our shiny new network hardware. Much like our US-West final improvement series maintenance in Irvine, we will be moving traffic in small batches which will minimize impact and allow us to move at a pace that will limit service interruption. Services impacted during the maintenance will include Web, E-Mail including Webmail, MySQL, the Web Panel and other services located in the US-West (Los Angeles) Data Center. You can check if you are affected by this maintenance by seeing where your services are hosted in the DreamHost Web Panel.

As we progress through the maintenance we will monitor comments here and provide you with immediate assistance.

Update March 20th 2:00AM: Due to downtime caused by our Irvine datacenter outage, we’ve opted to postpone this maintenance. Once we are able to reschedule this, we will re-post this notice! Thanks for your understanding!


 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
12:42 pm

Network Improvement Series Maintenance – US-West Data Center (Los Angeles, CA) – Core Router Software Upgrade – Wednesday March 13(RESOLVED)

Posted (February 25th, 2013 at 12:42 pm PST) by andrea

On Wednesday, 3/13 at 8pm, we will be performing a core router software upgrade in our US-West Data Center (Los Angeles, CA). We are scheduling a 4-hour maintenance window to safely test and monitor all the steps needed for this maintenance. During this upgrade, the network traffic will flow through temporary and thoroughly pre-tested routers, to minimize downtime.

This core router software upgrade is necessary to move on to the last step of the network maintenance in this data center, which has been previous postponed:
Network Improvement Series Maintenance – US-West Data Center (Los Angeles, CA) – Part 2/2 of Final Step (Postponed)
We will post an announcement once we are ready to go ahead with this final step again.

As always, please contact support if you have any questions. You may also check if you’re affected by this maintenance by looking up your data center location on your DreamHost panel.
We will update this status post as the maintenance moves forward and will be monitoring comments as well at that time.

Update March 14th 12:55am PDT: We have successfully completed the core router software upgrade. All network traffic is now going through the upgraded and tested routers. We will continue our network maintenance with the final step soon — look for an announcement on this status page!
As always, please contact support if you have any questions.

Previous updates available after the jump →


11:52 am

Network connectivity issues at US-West Data Center (Los Angeles, CA) – ( RESOLVED )

Posted (February 22nd, 2013 at 11:52 am PST) by brians

We are currently experiencing network connectivity issues in our US-West Data Center (Los Angeles, CA). This is resulting in site downtime. Our network engineers are working to restore connectivity. We will update this post with further information as it becomes available.

UPDATE February 22nd 12:45PM PST:: We are happy to report that this outage has been resolved. All services are up and fully functional. Thanks again for your continued patience.

Previous updates available after the jump →


12:20 pm

DNS Server Problems ( RESOLVED )

Posted (February 14th, 2013 at 12:20 pm PST) by brians

We are experiencing some service interruption as a result of an issue with our DNS servers. This may cause site downtime, slowness, etc. Email could be impacted as well. We are working quickly to resolve this and will provide an update as soon as possible.

UPDATE February 14th 2:04PM: The DNS servers remain stable, and we’re continuing to monitor but consider this incident resolved. Thanks again for your continued patience.

Previous updates available after the jump →


11:55 am

Connectivity issues with Web Panel and some services (RESOLVED)

Posted (February 12th, 2013 at 11:55 am PST) by patrickm

We apologize for the issues you may be having with accessing your Customer Web Panel at this time, as well as any access issues with WebMail or your hosting services. We are working to resolve an issue which is causing this loss of connection and we will be updating this post with additional information as soon as possible. Thanks for your continued patience.

Update 12:20pm – This issue has now been resolved! This incident caused connectivity issues with your Customer Web Panel, WebMail and some hosting services. Our network engineers have identified the cause and have put a fix in place to prevent this in the future. All services are back online, but please contact Support with any questions you might have! -PM


12:25 pm

Service connectivity issues for 2 racks of servers in our US-West Los Angeles datacenter (RESOLVED)

Posted (February 9th, 2013 at 12:25 pm PST) by patrickm

We are currently experiencing connectivity issues on 2 racks of servers in our Los Angeles datacenter. This is affecting all web services for the servers listed in the first comment of this status post. Our engineers are actively working to resolve this outage and we will be updating this post as we have new information.

Please contact Support from the Support > Contact Support section of your Web Panel if you have questions.

Update February 9th 1:38pm: Our network engineering team has given the all clear and we are no longer seeing any packet loss. Should you have any further issues with your sites being down, please contact our support team via the web panel by going to Support -> Contact Support. -Oscar Previous updates available after the jump →


7:02 am

Network Issues Between our Downtown Los Angeles and Irvine Datacenters (RESOLVED)

Posted (February 7th, 2013 at 7:02 am PST) by Justin@DH

We are experiencing packet loss in traffic between our Downtown Los Angeles and Irvine datacenters at this time. This is causing specific issues with customers with split webserver and mysql services between the two datacenters, and mysql servers in the irvine datacenter. Our network engineering team is online and looking into the issue currently. When we have further information we will update this post.

Update February 7th 7:50am PST: The issues appeared to be gone for a short time but have returned, we have all our admin and network engineering team on this issue at this time. Our apologies for the issues this is causing those customers affected.

Update February 7th 8:20am PST: Our network engineering team has been able to fix the issue, which appears to be related to a DDoS from specific sources. These have been corrected and further investigation for hardening of the network will continue. Our apologies for the issues you experienced this morning.


5:16 pm

Email Downtime for Homiemail-sub5 (RESOLVED)

Posted (January 31st, 2013 at 5:16 pm PST) by Dan J

We are currently experiencing a few problems with our homiemail-sub5 mail server, which our admin team is looking into. The outage should not result in any lost mail, however, you will be unable to access any email hosted on this server until this is resolved. This only affects any users on our homiemail-sub5 mail cluster. To check if you are affected, click the “Account Status” link on the top right of your DreamHost panel. Scroll to the bottom, and under “Your Email Server”, if it shows homiemail-sub5, this will explain any mail problems you are noticing at present. Please note, this only affects email hosted on homiemail-sub5. This does not affect any web or MySQL servers. We sincerely apologize for the inconvenience and thank you for your patience! We will post further updates below as they become available.

Update Jan. 31st 5:58pm PST: The file servers are back online now. Our admins report that the mail appears to be operating properly now and there should be no delay for any mail at this time. We are continuing the investigation to ensure this will not happen again but for now we are resolving this post. If you do continue to see any problems then please contact support through the DreamHost panel.

Previous updates available after the jump →


4:51 pm

Network Improvement Series Maintenance – US-West Data Center (Los Angeles, CA) – Part 2/2 of Final Step (Postponed)

Posted (January 31st, 2013 at 4:51 pm PST) by Ralph

We are excited to announce that we are at the end of our Network Improvement Series! This series of improvements started back in May is finally nearing completion. We started this project when we realized our network stability was in danger as we had ourgrown our existing hardware. We have invested heavily and painstakingly planned to make sure we can provide you with the most stable network possible. On Wednesday, February 6th we will be completely switching over to our new network infrastructure in the final data center US-West (Los Angeles, CA). This maintenance will start at approximately 8pm and the window is 4 hours. During this maintenance we will be moving public traffic to our newly installed network, putting all live traffic on our shiny new network hardware. Much like our US-West final improvement series maintenance in Irvine, we will be moving traffic in small batches which will minimize impact and allow us to move at a pace that will limit service interruption. Services impacted during the maintenance will include Web, E-Mail, MySQL and other services located in the US-West (Los Angeles) Data Center. You can check if you are affected by this maintenance by seeing where your services are hosted in the DreamHost Web Panel.

As we progress through the maintenance we will monitor comments here and provide you with immediate assistance.

Update February 7th 3:40am PST: Everything is back to normal on the old setup. We will reschedule this maintenance and post to let you know when. Thank you for your patience. We are confident we learned from tonights events in order to have a successful maintenance on our next attempt.

Previous updates available after the jump →


4:01 pm

Network Improvement Series Maintenance – US-West Data Center (Los Angeles, CA) Part 1/2 of Final Step – Wednesday January 30, 6pm (4 hours) (RESOLVED)

Posted (January 25th, 2013 at 4:01 pm PST) by Ralph

We are nearing the end of the Network Improvement Series that we started back in May of 2012. We are very pleased with the performance of the new hardware we have installed to replace our previous network set up. Wednesday’s maintenance, on January 30th, in our US-West Data Center (Los Angeles, CA) is not customer-facing, however, changes made in the DreamHost panel (modifying or creating domains or users, for instance) may be delayed. We are scheduling a wide four-hour window to complete the maintenance, pre and post testing, and to cover any unforeseen complications.

You can check if you’re potentially affected by this on the DreamHost panel.
If you have any questions about this before or during the process, please let us know by contacting Support. We will be happy to help. During the maintenance, we will also leave the comments open and will be monitoring them as we provide periodic updates during the maintenance window.

We apologize for any inconvenience this may cause and appreciate your continued patience while we further improve our infrastructure.

Update January 30th 9:20pm PST: We’re all set, this step of our maintenance plan has been executed and completed. Thanks to everyone that helped and provided feedback!

Previous updates available after the jump →


5:34 am

Mail downtime on homiemail-sub5 (RESOLVED)

Posted (January 25th, 2013 at 5:34 am PST) by chalise

We are currently experiencing a few problems with our homiemail-sub5 mail server, which our admin team is looking into. The outage should not result in any lost mail, however, you will be unable to access any email hosted on this server until this is resolved. This only affects any users on our homiemail-sub5 mail cluster. To check if you are affected, click the “Account Status” link on the top right of your DreamHost panel. Scroll to the bottom, and under “Your Email Server”, if it shows homiemail-sub5, this will explain any mail problems you are noticing at present. Please note, this only affects email hosted on homiemail-sub5. This does not affect any web or MySQL servers. We sincerely apologize for the inconvenience and thank you for your patience! We will post further updates below as they become available.

Update Jan 25th 7:35am: The switch has been replaced and all necessary configurations have been applied. Mail services for homiemail-sub5 are now online and are processing mail. We will continue to monitor this situation to ensure all services are working as expected. There is a delay in receiving email as the mail queues are processing incoming mail again, but should clear up within a few hours. We apologize for any inconvenience this caused and appreciate your patience while we resolved this outage. -Oscar Previous updates available after the jump →


 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
2:53 pm

WordPress 3.5.1 – Maintenance and Security Release Available (RESOLVED)

Posted (January 24th, 2013 at 2:53 pm PST) by mir

WordPress 3.5.1 is now available for new installs and updates from the DreamHost One-Click Install panel. Please be aware that this is a security release, and as such, it is highly recommended for all customers.

You can find the release notes on WordPress.org, and further details in the codex.

Automatic upgrades, for customers that have them enabled, will begin shortly. If you would like to see if you have automatic upgrades enabled, or want to run your own, you can do so directly from the DreamHost Panel.


4:53 pm

Network Improvement Series Maintenance – US-West Data Center (Irvine, CA) – Final Step – Wednesday January 23rd 6pm (4 hours) (RESOLVED)

Posted (January 22nd, 2013 at 4:53 pm PST) by Ralph

This Wednesday we will be finalizing our Network Improvement Series Maintenance on our US-West Data Center based in Irvine, California. The maintenance is set to start at 6pm on January 23rd and is scheduled to last approximately 4 hours. Once successfully complete, this will conclude our Network Improvement Series, which started back in May 2012, for the Irvine data center. During this maintenance, we will be moving public traffic from our old network switches to the newly installed switches. This will put all network traffic on our new network hardware. During the maintenance, we will be moving the public network in small batches. The plan we created will minimize the service impact from this maintenance as much as possible. As we progress through the maintenance we will monitor comments here and provide you with immediate assistance. We are excited to be concluding the Network Improvement Series in Irvine. Once completed we will be moving on to finalize our other US-West data center based in Los Angeles.

Update January 23rd 8:00pm PST: US-West Irvine, Ca data center maintenance complete! We now have this data center’s network completely working with our new upgraded hardware. This project started back in May, and for the Irvine data center, it is complete! This was an exciting step for us! On to our US-West Los Angeles, Ca data center next! That is however is for another day! We will keep you updated on our plans to fully complete the Network Improvement Series.

Previous updates available after the jump →


1:32 pm

Apache issues in US-West Data Center (Los Angeles, CA) (RESOLVED)

Posted (January 20th, 2013 at 1:32 pm PST) by danwy

We are currently experiencing some issues in our LAX datacenter that are resulting in down websites for a small portion of our customers. This is in one of our datacenters that we are in the process of clearing out, so the number of affected customers is relatively small though. Email, MySQL and central services such as the web panel are all unaffected. We’ve found the cause of the issue and are already working on a resolution to restore the websites affected by this issue. We will update this post with more information shortly.

Update Jan. 20th 2:00pm PST: Upon further investigation we’ve found that most of the services affected by this are nearly empty, so there shouldn’t be much (if any) customer facing downtime. We are in the process of re-configuring all of the affected services to correct the problem and restore service to any domains still on these services. We will update this post again shortly with the status of how these updates are going. Please do contact support if you feel you are having problems related to this and we’ll be happy to investigate it further for you.

Update Jan. 20th 2:20pm PST: The updates that we ran on the affected servers have cleared up any remaining issues and all affected Apache services are now up and running. If you continue to have problems with any sites hosted in this datacenter, please let us know and we’ll be happy to look into it for you.


10:22 am

Mail downtime with homiemail-sub5 and webmail (RESOLVED)

Posted (January 17th, 2013 at 10:22 am PST) by Oscar

We are currently experiencing problems on one of the file servers that the mail machines mount to. Because this is where the mail files are stored it is not possible to access customer mail while we get this corrected (mail will still be accepted for delivery and written to disk once the file server is back up). Our system administration team is diligently working towards a solution. Again, this only affects users who are hosted on homie-sub5. You can confirm whether you are affected by clicking the ” Account Status ” link here http://panel.dreamhost.com. Scroll to the bottom and you will see a field labeled ” Your Email Server “. If its set to homiemail-sub5, that would explain any mail problems that you are encountering. We sincerely apologize for any inconvenience this has caused and appreciate your continued patience. Please check back for future updates.

Update Jan. 17th 10:45am PST: This issue has now been resolved and you should be able to connect to your email via webmail or email client. There was no mail lost during this outage and we are currently reprocessing all messages received during this time. Thanks your patience while we resolved this issue. – Oscar


11:12 pm

Web Panel and Webmail Down (RESOLVED)

Posted (January 15th, 2013 at 11:12 pm PST) by Steven

I am sorry to report that we have run into trouble on our central database server. This is causing downtime for the Customer Web Panel, Webmail, and our internal systems. We are actively investigating this issue and hope to have it resolved very soon. Additional information will be posted as it becomes available, or when this has been resolved. Thanks for your patience as we work to get everything back online.

Update Jan. 15th 11:56pm PST: As of 11:40pm all services have been restored. Our admin team found the Panel and Webmail machines were experiencing connectivity issues. The were able to find the source of the problem and resolve it. No customer services (web, mysql, or email) should have been affected. This was isolated to just the Web Panel and Webmail services. Once again, we thank you for your patience and apologize for any trouble this may have caused. -Steve


4:46 pm

DreamHost control panel down (RESOLVED)

Posted (January 15th, 2013 at 4:46 pm PST) by Seohee

The DreamHost panel (https://panel.dreamhost.com) is currently unavailable. Our administrators are working on finding the cause, so we’ll make sure to provide you with further updates shortly.

We are sorry for the trouble, and appreciate your patience at this time.

Update Jan. 15th 5:07pm PST: The issue appears to have been due to a database lockup, which then resulted in a pile-up of connections from the panel machines. This pile-up then caused the panel machines to run out of memory. This has been corrected, and our administrators are working on putting a solution in place to prevent this from happening again in the future.

The panel is up and running well at this time. Again, we appreciate your patience while we got this worked out, and are sorry for the trouble this has caused. – Seohee

 


12:22 am

Network Outage In Our Irvine Data Center (RESOLVED)

Posted (January 15th, 2013 at 12:22 am PST) by Steven

We are experiencing a bit of a network flap in our Irvine data center. Our system administrators and network engineers are currently working to identify the cause and resolve it. We will update this post as soon as further details are available! Thank you for your patience.

If you are not sure which of our data centers you are in, you can look that up in your DreamHost panel under Status / Data Centers. Please note, this only affects customers located in Irvine.

Update 1/15 4:10am PST: All networks are functioning normally once again, and all sites should be back up and running as usual. We sincerely apologize for the length of time your sites were down; we were diligently working to restore connectivity as quickly as possible. We will post a follow-up later today with more details. Thanks once more for your patience. If you are still experiencing issues with your site, it is not related to this issue, so please contact support via Support / Contact Support in your DreamHost panel so we can help!

Previous updates available after the jump →


4:35 am

Web Panel and WebMail inaccessible (RESOLVED)

Posted (January 12th, 2013 at 4:35 am PST) by patrickm

We regret to post that we are having issues with our central database server, which is causing downtime for the Customer Web Panel, WebMail and our internal systems. We are actively investigating this issue and hope to have it resolved very soon. We will update this post with more details if we have them, or when the issue has been resolved. Thanks for your patience as we get everything back online.

Update 4:46am – We have resolved this problem and both the Customer Web Panel and WebMail access has been restored! Thank you for your patience as we made quick work of this outage! -PatrickM


4:19 pm

Network Maintenance in US-West Data Center (Irvine, CA) Wednesday January 16th 6pm (4 hrs) (RESOLVED)

Posted (January 11th, 2013 at 4:19 pm PST) by Ralph

As part of our continued effort to improve our network services, we are conducting hardware maintenance in our US-West Data Center (Irvine, CA) on Wednesday January 16th 6pm. We don’t expect any customer-facing service downtime, but changes made in the DreamHost panel (modifying or creating domains or users, for instance) may be delayed. We are scheduling a wide four-hour window to complete the maintenance and the pre and post testing and to cover any unforeseen complications, but the actual process will likely be much shorter.

You can check if you’re potentially affected by this on the DreamHost panel .
If you have any questions about this before or during the process, please let us know by contacting Support We will be happy to help. During the maintenance, we will also leave the comments open and will be monitoring them. We will also provide periodic updates during the maintenance window.

We apologize for any inconvenience this may cause and appreciate your continued patience while we further improve our infrastructure.

Update Jan. 16th 10:25pm PST: Our tests are complete, as is the maintenance! We appreciate your patience.

Previous updates available after the jump →


2:57 pm

Email Hardware Maintenance in US-West Data Center (Los Angeles, CA) Tuesday January 15th 8pm PST (4 hrs) (RESOLVED)

Posted (January 11th, 2013 at 2:57 pm PST) by Ralph

As part of our ongoing network improvements, we will be conducting email hardware maintenance in our US-West Data Center, Los Angeles on Tuesday, January 15th at 8:00pm PST. This will affect less than 10% of the email service customers in that location. We are scheduling a four-hour window to complete the maintenance, perform post maintenance testing and cover any unforeseen complications. The actual downtime to your email service is expected to be much shorter. During the maintenance we will be updating this post to provide you with the actual status of mail service throughout the maintenance. When service is interrupted, you will not be able to log in, check email, or send email, also any mail sent to you will most likely be queued by the sending machine. As soon as the maintenance is complete, and the sending machine tries to re-send the message, queued mail will be delivered successfully.
You can check if you’re potentially affected by this on the DreamHost panel.
If you have any questions about this before or during the process, please let us know by contacting Support. We will be happy to help. During the maintenance, we will be allowing commenting and will be assisting you here as best we can.

This post will be your best source for periodic updates and information regarding this maintenance window.

We apologize for any inconvenience this may be causing and appreciate your continued patience while we further improve our infrastructure.

Update Jan. 15th 08:57pm PST: The file servers are all now operational again. Everything is working properly so we are closing this maintenance window. If you have any problem or then please let us know by contacting Support. Thanks!

Previous updates available after the jump →


 
© 1996-2013, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
7:36 am

Network Maintenance for US-East Datacenter – Maintenance To Resolve Mysql Issues (RESOLVED)

Posted (December 17th, 2012 at 7:36 am PST) by Justin@DH

Our apologies for the issues with mysql connectivity in the US-East Datacenter this morning. Our onsite datacenter staff are performing an urgently needed maintenance to replace a defective network device. Staff are on site and we are expecting the maintenance to be completed shortly. We will update this post when we have further details on its completion.

Update Dec. 17th 7:37am PST: The maintenance has completed, the hardware has been replaced. We are watching the systems to make sure all is fully functional at this time.


12:47 pm

WordPress 3.5 Major Release Now Available! (RESOLVED)

Posted (December 11th, 2012 at 12:47 pm PST) by mir

The newest version of WordPress, 3.5, is now available on the DreamHost panel for new installs and upgrades. You can also upgrade from directly within your site’s WordPress admin panel, if you prefer.

This is both a feature and maintenance release, and is highly recommended for all customers! Its new features include a new and improved media workflow, new default theme, and a Tumblr importer

For more details, you can check out the release notes, here, with further details available in the codex.

If you run into issues after an upgrade, as a first step, take a look at this thread on 3.5 in the DreamHost forums, and the 3.5 master list at the WordPress.org forums, where issues relating to upgrades are being tracked.

The update will be automatically installed for customers who have automatic upgrades enabled. You can check your settings, and change these preferences directly, by visiting the DreamHost One-Click Panel. Once there, click “Manage Installed Applications,” select the domain, then look under the Upgrade Action column for details.

If after upgrading, you’re having issues with the ‘Add Media’ button and have Google Pagespeed enabled, check out this post for further information!


3:45 am

Mail delays with homiemail-sub4 mail server (RESOLVED)

Posted (November 16th, 2012 at 3:45 am PST) by patrickm

We are currently experiencing delays with mail being delivered from our homiemail-sub4 mail cluster. Our team has resolved the root issue which caused the problem, but delays of up to a few hours are being seen still at this time. Email from this cluster will be delivered, but may take more time than usual until the queues on the servers have been cleared out.

You can see if your mail accounts are on this cluster by logging in to your Web Panel and clicking on the “Account Status” drop-down in the upper right section of any page. Under “Your Email Cluster” will show “homiemail-sub4″.

Please feel free to contact Support (from the Support > Contact Support page in your Web Panel) if you have any questions or if you are experiencing any other issues!

Update Nov. 17th 6:43pm PST: The backlog of messages has been taken care of and this is now resolved!

Previous updates available after the jump →


12:24 pm

Network Maintenance for US-West Data Centers Saturday November 17th 6pm PST (6 hrs) (RESOLVED)

Posted (November 14th, 2012 at 12:24 pm PST) by andrea

Starting Saturday, November 17th at 6pm US Pacific Standard Time, we will be conducting planned maintenance on network circuits that connect our US-West data centers (Los Angeles, Irvine, and LAX) to each other. These circuits provide connectivity for overall network redundancy, and improve performance for customers with Web and MySQL services in multiple US-West data centers. To find out if you may be affected by this maintenance, go to the DreamHost panel main navigation and check under Status > Data Centers, and check if you have associated Web and MySQL services at different US-West data center locations.

The maintenance window is scheduled for 6 hours so that we can perform the upgrades across all of the relevant network systems, then test and confirm that it has been succesful. However any one affected customer (with services in two or more data centers) should not experience any more than up to 10-20 minutes of unavailability of services during the maintenance window, due to the way we’ve planned it out and will implement it to minimize customer impact.

We appreciate your understanding and patience while we conduct this maintenance. It is a necessary part of our Series of Network Improvements, and it’s a key step before we can complete our final phase core router upgrades.

Please note, this maintenance is dependent on the successful completion of the Hardware Replacement being performed on Wednesday and various pre-maintenance sanity checks. Once all of these checks are completed successfully, we will update this status post by Friday noon PST to confirm that it will proceed as planned.

We will also update this status post and open the comments once we start, and will provide periodic updates until conclusion. If you believe you are being affected by an unrelated service issue, please contact Support through the DreamHost Panel. We will also be monitoring Status Post comments throughout the maintenance.

Update: Nov. 17th 7:37pm PST: This maintenance window is now finished, with no downtime! We expected some, but were able to pull everything off with none! Thank you for your patience!

Previous updates available after the jump →


4:49 pm

Core Router Hardware Replacement for US-West Data Center (Los Angeles, CA) Wednesday November 14th, 8pm PST (4 hrs) (RESOLVED)

Posted (November 13th, 2012 at 4:49 pm PST) by Ralph

Wednesday November 14th, starting at 8pm Pacific Standard Time, we will be replacing hardware on our core router in one of our Los Angeles based data centers. There will be no service interruption during this maintenance, however we are scheduling a 4 hour window where we will be monitoring that the replacement goes smoothly. Once this maintenance starts, we will open the comments and will be watching closely to make sure it is completed successfully.

Update Nov. 14th 9:15pm PST: We’re all done with no downtime! We are closing this maintenance window now, as the internal hardware upgrade went as planned. This allows us to continue with our maintenance window scheduled for next Saturday, at 6pm.

Previous updates available after the jump →


2:44 pm

VPN service to be discontinued

Posted (November 7th, 2012 at 2:44 pm PST) by mir

For some time now, DreamHost has offered a VPN (Virtual Private Network) service to securely connect to DreamHost servers through our partner, CloudConnector. At the point that we introduced it, this service was useful for many of our customers, as most other technologies available at the time (such as Telnet and FTP) did not support encryption. Today, however, encrypted protocols such as SSH and SFTP are widely available, other superior VPN services are readily available, and the software required to connect to our VPN is not compatible with many newer operating systems. As a result, we have made the decision to phase out our VPN service.

If you are one of the approximately 250 customers who had active VPN service, we will be prorating your most recent charges for this service and applying them as a credit to your account — please feel free to contact DreamHost Support if you have any questions.


10:47 am

Intermittent outage in our US West LAX Data Center (RESOLVED)

Posted (November 6th, 2012 at 10:47 am PST) by Oscar

We experienced a brief network outage in our US West LAX Data Center, which caused web and MySQL downtime (email services were not impacted by this issue). Our system administrators identified the cause as a network “hiccup” in the core router which caused connectivity issues within our LAX datacenter. This outage lasted approximately 15 minutes and corrected itself while our Network Engineering team was investigating. We are, however, noticing issues between mysql machines communicating to web servers which are still causing sites not to load correctly. We are continuing to monitor all services and will make sure to update this post if we notice any further issues. We apologize for the trouble and appreciate your patience. Please contact support for any issues you have related to your site being down. You can do so by submitting a ticket through the DreamHost panel (https://panel.dreamhost.com/), under Support/Contact Support.

Update November 6th 2:30pm PST: Our Network Engineers have given the all clear and we are resolving this status post. All DNS queries are successfully resolving and we have not seen any further DNS/mysql issues. Should you have any further issues with your sites, please submit a support request and we will be glad to look into the issue for you. -Oscar Previous updates available after the jump →


8:50 pm

Core Router Software Update for US-West Data Center (Irvine, CA) Sunday November 11th 6pm PST (4 hrs) (RESOLVED)

Posted (November 5th, 2012 at 8:50 pm PST) by Seohee

Sunday November 11th, starting at 6pm Pacific Standard Time, we will be applying a manual software update on the core router of our US-West Data Center located in Irvine, CA. Customers with services in this data center will experience less than 20 minutes of downtime when the software update is being performed. You can find out what services you have at this data center by checking the Status->Data Centers section of the DreamHost Web Panel. This maintenance affects the following services:

Web, E-Mail, MySQL, DreamObjects, Dedicated, VPS

We thank you for your understanding and patience during this core router software maintenance.

We will be updating this status post right before the maintenance, then provide periodic updates through to the conclusion of the maintenance. We will be actively monitoring support submitted through the DreamHost Panel as well as comments posted here.

Update Nov. 11th 7:20pm PST: We are calling this maintenance completely resolved now. The software update has completed successfully and the issues with the final networks is resolved. We thank you for your patience and communication throughout the maintenance.

Previous updates available after the jump →


1:19 pm

Network Maintenance for US-West Data Centers Saturday November 3rd 8pm PDT (6 hrs) – (RESOLVED)

Posted (October 30th, 2012 at 1:19 pm PST) by andrea

Starting Saturday, November 3rd at 8pm US Pacific Daylight Time, we will be conducting planned maintenance on network circuits that connect our US-West data centers (Los Angeles, Irvine, and LAX) to each other. These circuits provide connectivity for overall network redundancy, and improve performance for customers with Web and MySQL services in multiple US-West data centers. To find out if you may be affected by this maintenance, go to the DreamHost panel main navigation and check under Status > Data Centers, and check if you have associated Web and MySQL services at different US-West data center locations.

The maintenance window is scheduled for 6 hours so that we can perform the upgrades across all of the relevant network systems, then test and confirm that it has been succesful. However any one affected customer (with services in two or more data centers) should not experience any more than up to 10-20 minutes of unavailability of services during the maintenance window, due to the way we’ve planned it out and will implement it to minimize customer impact.

We appreciate your understanding and patience while we conduct this maintenance. It is a necessary part of our Series of Network Improvements, and it’s a key step before we can complete our final phase core router upgrades.

We will update this status post once we start and provide periodic updates until conclusion. If you believe you are being affected by an unrelated service issue, please contact Support through the DreamHost Panel. We will also be monitoring Status Post comments throughout the maintenance.

Update November 19th 11:20am PDT: This maintenance has been conducted and completed without downtime on 11/17th, so it’s considered resolved. Please see more details at Network Maintenance for US-West Data Centers Saturday November 17th 6pm PST (6 hrs) (RESOLVED)

Update November 3rd 12:15pm PDT: Tonight’s maintenance has been postponed. We will repost once we are ready to commit to a new date for this maintenance to occur.


4:33 pm

US-East Region Storm Preparedness (RESOLVED)

Posted (October 29th, 2012 at 4:33 pm PST) by andrea

As many of you who are based in the US are aware, there is a major storm system impacting the East Coast today, with potential impact to Virginia where our US-East region data center is located. This notification is to advise that DreamHost is well prepared to handle any power outages or other issues that may arise during the storm and its aftermath.

At this time, we have very few customers provisioned in our US-East region data center, because it was only brought online in August 2012. So the vast majority of customers will be entirely unaffected by any issues related to the storm. You can find out where your services are located by logging into the Panel and clicking “Status” under the main menu on the left, then “Data Centers”.

For those few customers who have services in US-East (Ashburn, Virginia), we want you to know that we do not expect any service issues or outages during the storm or its aftermath, because we have layers of redundancy in place for all major systems, as follows:

• US-East is on two entirely separate power grids, so a failure of one grid will not impact services;

• if both power grids go down, we have sophisticated battery and generator backup systems (uninterruptible power supplies) in place that will keep the data center online for 3 days before the generators require fueling, and indefinitely with generator fueling arrangements in place;

• there are generator-refueling contracts in place with a number of suppliers to ensure we can run the data center indefinitely on generators;

• our network connectivity to US-East is protected and redundant; and

• the building is rated for major storms and will not be impacted.

We will update this status post if any issues occur at US-East, but we do not expect any during this storm.

Update: Wednesday October 31sth 5:30pm PDT: We are happy to report that we’re in the clear. We were not affected at all, and it’s no longer necessary to continue monitoring the situation. We are resolving this post, but feel free to contact support if you have any questions.

Previous updates available after the jump →


 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
11:01 am

Proactive Security Maintenance (New Login Keys)

Posted (October 24th, 2012 at 11:01 am PST) by mir

This is just a heads-up for everyone that you will likely see a key change warning the next time that you are logging in with your shell user. This is not a cause for concern as it is the result of a security maintenance which we are performing to prevent exploitation of weak or outdated keys (the maintenance should completed in the next 24 hours). The resulting warning will look like this (but will show your user and the server key which we have removed from the example):

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the RSA host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
REMOVED FOR SECURITY REASONS
Please contact your system administrator.
Add correct host key in /home/USER/.ssh/known_hosts to get rid of this message.
Offending key in /home/USER/.ssh/known_hosts:10278
Password authentication is disabled to avoid man-in-the-middle attacks.
Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.

To get rid of the warning, there are three methods. Please keep in mind that the modification to fix this needs to be performed on the client side, so if you are getting this connecting from home/work, it is that computer that needs to be updated.

The first method, you can delete your entire known_hosts file if you have several hosts in need of updating. On first login it will confirm the new key.

The second method, you could run in your shell a command like “ssh-keygen -R HOSTNAME”.

The third method, the known_hosts file will need to be edited. In the above case, one would open the file /home/USER/.ssh/known_hosts in their preferred text editor, and remove the problem line.

The error message lists the line number; it can be found after the file name in the error. In the above example the line number would be 10,278. If you are using the “vi” editor, you could type “:10278″ without quotes to get to that line, then hit the “d” key twice to delete it. Other editors should have similar features to assist you in removing the line of text.

(If you have accessed your server from any other computers before this key change, the same change will need to be done there on your next ssh or sftp login to the server.)

That should cover the situation but please let our support team know if you have any questions or concerns and thank you for choosing DreamHost!

 

Update: Tuesday October 30th 1:33pm PDT:

 

We’ve got a brand-spanking new panel tab now available, which can be used to look up the correct keys for all machines on your account! You can access this by clicking “Users” > “SSH Keys”, or by following this link!


4:28 pm

Core Router Software Upgrade – US – West Data Center (Los Angeles, CA) Sunday October 21st 8pm PDT (4 hours) POSTPONED

Posted (October 19th, 2012 at 4:28 pm PST) by Ralph

This Sunday, October 21st at 8pm PDT, we will be performing a software upgrade in our US West Data Center based in Los Angeles, CA, in a 4 hour maintenance window. This upgrade was already successfully applied to our US West – Irvine, CA Data Center on October 3rd. The successful upgrade did not effect customer’s services, so we are confident this current upgrade will come with similar results. There are redundant routing engines in place which will allow us to minimize any service interrupting impact. Each engine can be upgraded individually in order to keep service active during the maintenance. We will update this status post once the maintenance starts and provide periodic updates until conclusion.

You can now view your service data center locations in the DreamHost Web Panel under Status->Data Centers.

Update: Sunday October 21st 11:10pm PDT: The rollback has completed, and all systems are up and running. After we’ve had a chance to analyze logs and data we’ve retrieved during this process, we will reschedule the core router software upgrade for another time. We will announce in a new status post as soon as possible. Thanks, once again, for your patience!

Previous updates available after the jump →


7:58 pm

Upgrade to Roundcube Version Monday October 8th 8pm PDT (1 hr) (RESOLVED)

Posted (October 8th, 2012 at 7:58 pm PST) by Dan J

We will be performing a software upgrade to the Roundcube installation to version 0.8.2. We estimate the upgrade to take around one hour. During this maintenance customers may experience intermittent outages or loss of session. We will of course keep you apprised of our progress as we work our way through the maintenance window.

Update: Monday October 8th 8:59pm PDT: The upgrade is now complete. If you notice any problems with the new version then please contact support through the Dreamhost panel. Thanks for your patience.

Previous updates available after the jump →


12:14 pm

Core Router Software Upgrades (West Coast Data Centers) Thursday October 4th 8pm PDT (6 hrs) POSTPONED

Posted (October 3rd, 2012 at 12:14 pm PST) by andrea

We will be performing a software upgrade which was recently made available on Thursday, October 4th starting at 8pm PDT. The window will last for 6 hours and will allow us enough time to upgrade both of our West Coast Data Centers.  This software upgrade is necessary to complete prior to the switch over to our new Core Router system.  We do not anticipate any service interruption during the upgrade as there are redundant routing engines in place. We are able to upgrade our non-active engine and then bring it online to take over while the others can be upgraded while they are inactive. We will of course keep you apprised of our progress as we work our way through the maintenance window.

 

Update: Thursday October 11th 4:50m PDT: To clarify, the router upgrade in one of the data centers was successfully completed. The other one is scheduled to happen soon. We’ll add a new status post as soon as we’re ready to go. As always, thank you for your continued patience while we’re improving our network infrastructure.

 

Previous updates available after the jump →


3:18 pm

Network Upgrade (West Coast Datacenter) – Redundant Internet Feed Upgrade Completed

Posted (September 27th, 2012 at 3:18 pm PST) by andrea

We are happy to report that we have completed the installation of an additional 10 gigabit internet feed in one of our our west-coast datacenters. This will further improve redundancy on our existing 10 gigabit lines. There was no negative impact during the installation, as the primary network was not affected. You should be noticing improved speeds on some of your services, so we wanted to share the good news.

Thank you for your continued patience while we further improve our network infrastructure.


4:10 pm

Network Upgrade (West Coast Datacenter) – Redundant Internet Feed Upgrade – September 20th, 11am PDT (2 hrs) (POSTPONED)

Posted (September 17th, 2012 at 4:10 pm PST) by andrea

This Thursday, September 20th, we will be installing an additional 10 gigabit internet feed in one of our our west-coast datacenters. This will further improve redundancy on our existing 10 gigabit lines. We are not anticipating any negative impact during the installation, as the primary network is not affected. After the install, you should be noticing improved speeds on some of your services.

Thank you for your continued patience while we further improve our network infrastructure.

Update: Monday September 24th 10:50am PDT:  Apologies for the lack of updates. This maintenance has been postponed, and we’ll post the new information as soon as we have it.


1:51 pm

Mail System Relocation Maintenance Sunday September 16th, 12am PDT (RESOLVED)

Posted (September 13th, 2012 at 1:51 pm PST) by Ralph

On September 16th, at midnight (00:00:00), the mail servers in our LAX Datacenter will be transferred to our Los Angeles Datacenter. This relocation will also upgrade our mail service to our freshly hardened and rebuilt network. Since this is hardware relocation, we will be taking mail services offline at approximately 12am Sunday September 16th PDT. We plan to have service restored by 2am PDT and will use the remaining window to make sure that everything is working as expected. During the move, you will not be able to send or receive email. Any incoming mail should be queue’d on the senders’ server and delivered later. However, please note that delivery behavior is dependent on the sending servers configuration, so some mail may bounce back to the outside sender. We thank you for your patience during this relocation and apologize for the inconvenience it may cause. This status post will be updated 15 minutes prior to start of the maintenance, and we will provide you with periodic updates as we progress through the maintenance window.

Update: Sunday September 16th 8:15am PDT: Our admins were able to find an issue with the new cluster quickly and have resolved the issue causing login errors. We’ve done some quick spot checking of accounts and they all seem to be working correctly now. We will now be going through support requests send in regarding this issue and will be doing further testing on those messages. If you are still having problems with your mail, please contact support so that we can look into your issue right away.

Previous updates available after the jump →


2:42 pm

WordPress 3.4.2 Maintenance and Security Release (RESOLVED)

Posted (September 7th, 2012 at 2:42 pm PST) by mir

The latest-and-greatest WordPress release, 3.4.2 is available for customers to manually install or upgrade from the DreamHost Panel.

You can also upgrade from directly within your site’s WordPress admin panel, if you prefer.

The update will be automatically installed for customers who have automatic upgrades enabled. You can check your settings, and change these preferences directly, by visiting the DreamHost One-Click Panel. Once there, click “Manage Installed Applications,” select the domain, then look under the Upgrade Action column for details.

For more information on the release, check out the announcement post and additional details on the codex.

Known issue:
Custom Field ‘update’ button fails to work properly this can be fixed through using the official Hotfix Plugin.

Fun Fact: WordPress 3.4.2 was tested and live on the DreamHost panel for customers within 15 minutes of release at WordPress.org


1:15 pm

Intermittent outage in one of our Los Angeles data centers (RESOLVED)

Posted (September 6th, 2012 at 1:15 pm PST) by danwy

We are currently experiencing a network outage in our Los Angeles area datacenters, which is causing web and MySQL downtime (email services are not impacted by this issue). Our administrators have identified the cause as network connectivity issues within our network between data centers. What this means in practical terms is that while all services are up and running, a handful of customers who have web and MySQL services in different data centers are experiencing outages as the components are unable to communicate with each other. We will make sure to update this post as soon as we have further information. We apologize for the trouble and appreciate your patience.

Update: Friday September 7th 1:00am PDT: All services and connectivity have been restored, and we will be following up with our vendor to prevent any further issues. Thank you, once again, for your patience while we worked toward a resolution.

Previous updates available after the jump →


6:38 pm

Hanjin server offline for hardware replacement (RESOLVED)

Posted (September 3rd, 2012 at 6:38 pm PST) by Dan J

Our admin team have taken the hanjin.dreamhost.com server offline temporarily. They are replacing some of the hardware components used with this server due to a failure. The server will remain offline until tomorrow morning Sept 4th and backup users will not be accessible until then. No data should be lost due to this maintenance. We apologize for this unscheduled maintenance and will continue to keep you updated on the situation.

Update: Friday September 7th 1:55pm PDT: The server is now back online and functioning correctly. We did some testing and can confirm that file access and permissions are working as expected. We are keeping this status post unresolved for a little while longer to keep an eye on the service. Thank you for your patience while we make sure this service continues to work adequately.

Previous updates available after the jump →


 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
5:40 pm

Border/Core Router Software Upgrades (Los Angeles Data Center) Thursday August 23rd 8pm PDT (4 hrs) (RESOLVED)

Posted (August 22nd, 2012 at 5:40 pm PST) by andrea

After successfully completing the Border/Core Router upgrade at our Orange County and LAX Data Centers, we’re moving on to the Los Angeles Data Center. The maintenance window will start at 8pm and last for 4 hours on Thursday, August 23rd. There is no expected service interruption during this upgrade as there are redundant routing engines in place >on the routers that will help us avoid service interruption. This process functions by having the non-active engine upgraded first, then brought online so the others can be upgraded while inactive. We will keep you updated on the progress as we begin this maintenance window.

If you have any service issues before the maintenance window, please contact support for assistance through the DreamHost panel.

Update: Monday August 23rd 9:15pm PDT:  Maintenance has completed with no downtime! Tomorrow, we’ll be performing more of the same to get our network in tip top shape!

Previous updates available after the jump →


6:16 pm

Network maintenance – component replacement and software upgrade (Los Angeles Data Center) Friday, August 24th 8pm PDT (4hrs) (RESOLVED)

Posted (August 21st, 2012 at 6:16 pm PST) by andrea

In accordance with our current network maintenance philosophy, we have been keeping a close eye on our newly installed systems in all our datacenters. In our smallest one, we discovered a component which doesn’t appear to function as expected. While it’s not causing any service issues, we decided to get it checked out, just in case. We are working with the vendor to ensure seamless replacement and software upgrade of that component. We set our maintenance window to 4 hrs, but we anticipate no more than 15 minutes of actual downtime. We will be monitoring the comments during the maintenance, and will keep you updated about the progress.

If you have any service issues before the maintenance window, please contact support for assistance through the DreamHost panel.

Update: Friday August 24th 8:17pm PDT: Maintenance has completed with no downtime, and in record time! Everything is showing green on our end, and that concludes tonight’s updates to our network topology.

Previous updates available after the jump →


3:23 pm

Border/Core Router Software Upgrades (Orange County Data Center) Monday August 20th 8pm-Midnight (RESOLVED)

Posted (August 17th, 2012 at 3:23 pm PST) by Ralph

This coming Monday, August 20th, we will be upgrading the software on our Border and Core routers at our Orange County Data Center. The maintenance window will start at 8pm and last for 4 hours. There is no expected service interruption during this upgrade as there are redundant routing engines in place on the routers that will help us avoid service interruption. This process functions by having the non-active engine upgraded first, then brought online so the others can be upgraded while inactive. We will keep you updated on the progress as we begin this maintenance window.

Update: Monday August 20th 11:48pm PDT:  We have successfully completed this hardware upgrade, without service interruptions. We are continuing the network maintenance series in our other data centers, so check back here for upcoming status posts. Thank you for your patience while we’re continuing to improve our services!

Previous updates available after the jump →


5:31 pm

Network Improvement Series Maintenance – Core Upgrade Part 3 (Los Angeles Data Center) – Friday August 17th 8pm PDT (4 hrs) (RESOLVED)

Posted (August 15th, 2012 at 5:31 pm PST) by andrea

Our Network Improvement Series Maintenance will continue on Friday, August 17th, with an adjustment to our core router configuration. Due to topology changes, we are removing a piece of hardware that is no longer needed in the routing process. We completed the required testing and staging but will be ready to roll back the changes if necessary.

No customer-facing service is connected to that hardware, we just wanted to keep you updated about our network maintenance progress.

We will update this status post throughout the maintenance and will be monitoring comments. If you have any service issues before the scheduled time of the maintenance, please contact support for assistance through the DreamHost panel.

Friday August 17th 8:25pm PDT:  Quick and easy! As we expected, no customer facing downtime occurred, and we’re all done!

Previous updates available after the jump →


4:51 pm

Upgrading web server email relay machine – August 20th, 12pm PDT (4hr) (RESOLVED)

Posted (August 15th, 2012 at 4:51 pm PST) by andrea

Monday, August 20th, starting at 12pm PDT, we will be upgrading the server that relays email generated on web servers. This includes, but not limited to, emails sent through phpmail and mail forms used on web sites.

We’re setting a four-hour maintenance window, but we anticipate only a very short delay in mail delivery while we move the services to a better server. No mail will be lost, only slightly delayed. This maintenance will not affect incoming mail or mail sent through email clients (including webmail) or other services, such as web or MySQL.

We will update this status post throughout the maintenance and will be monitoring comments. If you have any service issues before the scheduled time of the maintenance, please contact support for assistance through the DreamHost panel.

Update: Monday August 20th 8:48pm PDT: The maintenance is now complete. Thanks everyone for your patience.

Previous updates available after the jump →


2:08 pm

Connection issues on some of our mail servers

Posted (August 6th, 2012 at 2:08 pm PST) by danwy

We are currently experiencing some issues with one of our mail load balancers. This is a machine that routes traffic (emails) through our mail clusters and as a result this is causing some disruption of mail and webmail services. Details are still developing at the moment and our admin team is already hard at work on the problem. If you are having issues sending or receiving mail at this time, it is likely related to this matter so please give us some time to get it resolved before continued testing. We will update this status post as soon as we have any information and thank you for your patience.

Update: Thursday August 6th 2:30pm PDT: The load balancer being investigated is currently under very heavy load which is leading to service disruption. Our admin team is in the process of identifying and correcting the source of the abnormal workload being put on this machine. We’ll update this post again as more information becomes available.

Update: Thursday August 6th 3:00pm PDT: We have resolved this issue and mail appears to be functioning normally again. Our admin team is still correcting a configuration issue which prevented our redundant measures from properly executing but the service disruption itself is resolved. We apologize for any inconvenience this caused you and thank you for your patience.


4:46 pm

Network Improvement Series Maintenance – Core Upgrade Part 2 (Los Angeles Data Center) – Thursday August 2nd 8pm PDT (6 hours) (RESOLVED)

Posted (July 31st, 2012 at 4:46 pm PST) by Ralph

Our Network Improvement Series Maintenance will continue on Thursday, August 2nd with the remainder of our upgrades. We have roughly five more racks left to switch over. The upgrade on Friday was successful. These five racks were not completed with the Friday maintenance as we did not want to push the maintenance window much further than we had already posted. We were able to minimize any downtime experienced to less than two minutes each rack. Our expectations are the same as last Friday, which are listed below.

The maintenance on Thursday will switch over specific subsets of our data center. This means that the move will happen in small chunks rather than the full data center at one time to minimize the impact on live services. In our previous upgrade we were able to limit downtime to these subsets to under two minutes each. If service interruption for a specific subset exceeds five minutes we will roll back the changes.

When the maintenance starts we will be monitoring comments and providing you with periodic updates as we progress through our process.

Update Thursday August 3rd 12:15am PDT: We have successfully completed the network maintenance! We had a slight bump in the process, but we were able to work out the issues relatively quickly and we are confident that the problem will not return. This is a huge step for us as we wind down our Network Maintenance Series and we thank you for your patience throughout. This maintenance is now considered resolved, so please Contact Support to report any issues you encounter with your services, as they are unrelated and should be investigated.

Previous updates available after the jump →


5:36 pm

Network Outage in LAX Datacenter (RESOLVED)

Posted (July 30th, 2012 at 5:36 pm PST) by Seohee

We are currently experiencing a network outage in our LAX datacenter, which is causing web, MySQL, and email downtime. Our administrators are working on identifying the cause. We will make sure to update this post as soon as we have further information. We apologize for the trouble and appreciate your patience.

Update July 31th, 2012, 1:46pm PDT:  Thank you for your patience as we investigated the cause of the outage that affected your services last night. Our admins have completed their analysis of the incident and have determined that it was due to a hardware failure in one of our routing devices. This resulted in a loss of connectivity between our LAX datacenter and our bandwidth providers. We rebooted the machine that was having issues and that restored connectivity to the affected networks. Total downtime was just over an hour.

We have already contacted our hardware provider and we will be getting a brand new router to replace this one with. As soon as we have the new hardware, we will be swapping out the router that failed. If we expect any downtime as a result of the swap, we will post a maintenance window on our status page in the near future. Again, we apologize for any inconvenience this outage caused for you. If there’s anything else we can do for you, please don’t hesitate to contact support and let us know.

Previous updates available after the jump →


6:03 am

Power Issues in Los Angeles Datacenter (RESOLVED)

Posted (July 30th, 2012 at 6:03 am PST) by chalise

We are currently experiencing issues with power strips in two of our rows in one of our Los Angeles datacenters. This is affecting a subset of our shared and VPS host servers. Our team is currently looking into this and working to restore connectivity as quickly as possible. We will update this post with further information as it becomes available.

Update July 30th, 2012, 6:15am PDT: Our datacenter team was able to locate and resolve the issue. All sites and services should be back up and running now! Thanks for your patience.


5:17 pm

Network Improvement Series Maintenance – Finalizing Core Upgrade (Los Angeles Data Center) – Friday July 27th 8pm PDT (6 hours) (RESOLVED)

Posted (July 25th, 2012 at 5:17 pm PST) by Ralph

Our Network Improvement Series Maintenance will continue on Friday with the cut over of the physical connectivity switch from the old hardware to the new. We are excited to get this completed. This step is the last major step in our Network Improvement Series Maintenance. We are seeing improved performance and network reliability as a result of the Series Maintenance though we still have some optimization to complete.

The maintenance on Friday will switch over specific subsets of our data center. This means that the move will happen in small chunks rather than the full data center at one time to minimize the impact on live services. In our previous upgrade we were able to limit downtime to these subsets to under 2 minutes each. If service interruption for a specific subset exceeds 5 minutes we will roll back the changes.

When the maintenance starts we will be monitoring your comments and providing you with periodic updates as we progress through our process.

Update Saturday July 28th 1:50am PDT: At this time, we’re concluding the maintenance window. We didn’t get to everything we’d planned on, however we were able to take care of about 70% of the work. We did this with very little service interruption over the entire 6 hour window. At this time, we’ll make a separate announcement, detailing our plans to finish up the work we’ve yet to complete. We thank you for your patience during this window!

Previous updates available after the jump →


 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
12:06 pm

Network Upgrade (West Coast Data Centers) – Redundant Internet Feed Upgrade – 11am PDT (2 hrs) July 26th (RESOLVED)

Posted (July 25th, 2012 at 12:06 pm PST) by andrea

This Thursday, July 26th, we will be installing a backup 10 gigabit internet feed in our west-coast datacenters. This installation will be adding a new internet feed in both locations. This will further improve redundancy on our primary 10 gigabit lines in the event of failure. Since this change will not be directly affecting our production network there is no anticipated customer impact. We just wanted to share the great news of further data center redundancy.


2:55 pm

Network Upgrade (West Coast Data Centers) – Redundant Internet Feed Upgrade – 10am PDT (2 hrs) July 25th (POSTPONED)

Posted (July 23rd, 2012 at 2:55 pm PST) by andrea

This Wednesday, July 25th, we will be installing a backup 10 gigabit internet feed in our west-coast datacenters. This installation will be adding a new internet feed in both locations. This will further improve redundancy on our primary 10 gigabit lines in the event of failure. Since this change will not be directly affecting our production network there is no anticipated customer impact. We just wanted to share the great news of further data center redundancy.

Update July 23, 11:42am PDT: Unfortunately, we were unable to finalize some of the necessary pre-testing for this maintenance. We are rescheduling for tomorrow morning, see new status post above. As a reminder, there is no anticipated impact to your services.


7:56 am

Network Issue in Irvine Datacenter (RESOLVED)

Posted (July 23rd, 2012 at 7:56 am PST) by Justin@DH

We are currently experiencing network connectivity issues with our internal private network of our irvine data center. The internal network is responsible for dns and configuration of services from our control panel. We are experiencing issues on some but not all shared mysql services as a result of the dns issues. This is resulting in site downtime for a small subset of our dedicated, VPS and shared hosting customers. Our network engineers and system administration team are working to restore connectivity at this time. We apologize to those customers who are affected and will update this post with further information as it becomes available.

Update July 23, 9:30am PDT: All services have been reporting OK for the last 40 minutes, so we are going to go ahead and resolve this post. The cause of the outage, a specific DDoS attack, has been found and will be analyzed for better protection from this newer kind of attack. If you are still seeing any issues with your service, it may be unrelated to this issue so please contact support and we would be happy to help.

Previous updates available after the jump →


9:51 pm

Network Improvement Series Maintenance – Central Service Router Upgrade Monday July 23 – 8pm PDT(4 hours) (RESOLVED)

Posted (July 19th, 2012 at 9:51 pm PST) by Brian@dreamhost

On Monday, July 23rd, we’ll be upgrading the network hardware that is responsible for our central web panel (https://panel.dreamhost.com) and all web based email service (http://webmail.dreamhost.com and http://webmail.yourowndomain.com) as well as our sign up and contact forms.

Related to the outbound DDOS that affected the firewall pair at one of our Southern California datacenters last week, we will be upgrading that infrastructure to be able to better handle spikes in traffic in the future. We will be performing as much of the configuration staging as possible before the physical cutover so there should be very little disruption outside of that brief period.

We anticipate a 4 hour window, and expect some downtime for the afore-mentioned services. Rest assured that website uptime, standard email delivery and IMAP/POP/SMTP will not be affected by this window.

If you have any questions or concerns, please contact support with the details, so we can assist you!

Update July 23, 11pm PDT: We have successfully completed the maintenance, quicker than expected and with no significant downtime. Our central services are completely transferred over to the new hardware which has improved specs and redundancy.

We now consider this case resolved. If you have any questions or concerns, please contact support with the details, so we can assist you!

Previous updates available after the jump →


8:03 pm

Network Upgrade (Orange County Data Center) – Redundant Internet Feed Upgrade – 10am PDT July 13th (POSTPONED)

Posted (July 12th, 2012 at 8:03 pm PST) by Ralph

This Friday, July 13th we will be installing a backup 10 gigabit internet feed. This installation will be adding an additional new internet feed to the Orange County data center. The reason for this is to further improve redundancy on our primary 10 gigabit line in the event of failure. Since this change will not be directly affecting our production network there is no anticipated customer impact. We just wanted to share the news of further data center redundancy.

Update July 13, 4:09pm PDT: We’ve had to postpone this upgrade to our network due to peering issues on our provider’s side of things. We’ll create another post, when this has been rescheduled. –Brian H


3:58 pm

Network Improvement Series Maintenance #5: Core Router Upgrade (Los Angeles) Monday, July 16 – 4pm PDT (8 hours) (RESOLVED)

Posted (July 12th, 2012 at 3:58 pm PST) by Ralph

In the past month, we have made significant progress with our network upgrade project. Due to these successes and our ability to minimize downtime we are confident that we can proceed with the next step of upgrading our core router at our Los Angeles-based data center.

The successful Orange County upgrade helped us find a way to avoid affecting customer services as much as possible. Since our confidence is high, and we anticipate no material customer impact we are starting our maintenance window a bit earlier than usual, at 4pm PDT.

The window will last 8 hours, until midnight. The Los Angeles data center is larger than Orange County, but the changes are the same and our new streamlined process has been extremely effective.

During the maintenance, we will continue to update you on the progress we are making. If you are experiencing problems prior to this maintenance window please contact support as your issue will not be related. During the window we will be monitoring support messages and status comments to ensure that customer impact is limited as much as possible.

Update July 17th, 12:05am PDT: This network maintenance is now successfully completed. Overall, everything went well, however, we did have 2 hours of spotty or intermittent mysql coverage for a group of servers. That should now be resolved, so there should be no lingering issues from this maintenance window. We appreciate your patience with this, if you are experiencing any issues, please contact support with the details, so we can assist you.

Previous updates available after the jump →


11:11 am

No connection to DreamHost panel (RESOLVED)

Posted (July 11th, 2012 at 11:11 am PST) by andrea

We are currently having connection issues to the DreamHost control panel and other central web-based services (contact form, sign up and Webmail) – the good news is that this means that customer websites are not impacted. Our teams are working on getting these working again, and we apologize for the inconvenience this is causing. We’ll update this post again shortly as soon as more information is available.

Update July 11, 6:40pm PDT: All issues related to this status post have been addressed so we are setting it to resolved.

Previous updates available after the jump →


12:12 pm

Webmail Database Issues (RESOLVED)

Posted (July 9th, 2012 at 12:12 pm PST) by Justin@DH

We are experiencing issues with the database connection between the webmail machines and the central mysql services. Our admins are already on the case and are working on repairing this. Until this is resolved, webmail will be showing an error connecting to database and will not be functional. When we have further updates on the progress of this fix, we will update this post. Our apologies for the issues affecting your webmail service.

Update July 11, 7:43pm PDT: Webmail is up and running again since our panel (and web-based services) outage today was resolved. We sincerely appreciate your patience while we got this worked out.

Previous updates available after the jump →


2:09 pm

Network issue in LAX datacenter (RESOLVED)

Posted (July 5th, 2012 at 2:09 pm PST) by Oscar

We are currently experiencing network connectivity issues in our LAX data center. This is resulting in site downtime for a small subset of our VPS and shared hosting customers. Our network engineers and system administration team are working to restore connectivity at this time. We apologize to those customers who are affected and will update this post with further information as it becomes available.

Update July 5, 2:28pm PDT: We experienced a brief period of degraded service resulting from  a DDoS that has been mitigated with the help of our new network hardware. Currently all services are back up and running so this matter is now resolved.


12:56 am

Network Improvement Series Maintenance – Non-Customer-Facing Hardware Upgrade (Orange County) Monday July 2 – 11am PDT(4 hours) (RESOLVED)

Posted (July 1st, 2012 at 12:56 am PST) by andrea

We are continuing with our network improvements by upgrading some more hardware in our Orange County datacenter. This upgrade should not affect customer services, this is just a notice to announce our next step in the Network Improvement Series Maintenance.

If you have any questions or concerns, please contact support with the details, so we can assist you.

Update July 2, 3:14pm PDT: The above maintenance has been successfully completed. Please keep an eye on dreamhoststatus.com, as we’ll be announcing our next steps in our Network Improvement Series Maintenance.

Previous updates available after the jump →


 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
4:05 pm

Network Improvement Series Maintenance #4: Core Router Upgrade (Orange County) Saturday, June 30 – 8pm PDT(4 hours) Part 3: Hardware Cut Over (RESOLVED)

Posted (June 29th, 2012 at 4:05 pm PST) by andrea

Last night, we successfully completed the software-related changes (Part 2 of maintenance #4) necessary for the core router upgrade in our Orange County datacenter. On Saturday, we are performing the actual physical upgrade, Part 3.

The process will involve moving cabling of individual subsets of our network in that datacenter, one at a time. Web and MySQL service interruption will occur while the individual networks are moved, but will only affect customers on those specific networks for a short time (< 5 minutes). We do have several physical connections to switch so we expect to use the entire four hour window.

We appreciate your patience while we work towards completing our network upgrades.

If you’re experiencing downtime *before* the window referenced above, please contact support with the details, as it’s unrelated to this maintenance. During the maintenance, we will keep you updated, so please check back here often.

Update July 1, 12:40am PDT: We have successfully completed the hardware upgrade for our Orange County datacenter core routers. This concludes maintenance #4. We will be continuing with our network improvements, so please keep checking back for upcoming maintenance notices.

If you have any questions or concerns, please contact support with the details, so we can assist you.

Previous updates available after the jump →


5:28 pm

Network Improvement Series Maintenance #4: Core Router Upgrade (Orange County) Thursday, June 28 – 10pm PDT(5 hours) Part 2: Cut Over — RESOLVED

Posted (June 27th, 2012 at 5:28 pm PST) by andrea

Part 2: Preparation to cut over to new hardware – Thursday, June 28th – 10pm PDT – 2am PDT ( 4 hour window). With the staging completed a while back, we will now proceed to do the necessary software preparation to cut over to the new core router hardware and use it in our live network environment. Web and mysql service interruption will occur during this process, but we estimate it to be limited to mere ‘blips’ at a time. We will need the full four hour window to complete the preparation.

If you’re experiencing downtime *before* the window referenced above, please contact support with the details, as it’s unrelated to this maintenance. During the maintenance, we will keep you updated, so please check back here often.

Part 3 will be the actual physical cut over to the new core router hardware itself. The process will involve moving cabling of individual subsets of our network in that datacenter, one at a time. Service interruption will occur while the individual networks are moved, but will only affect customers on those specific networks for a short time (< 5 minutes). We do have several physical connections to switch so we expect to use the entire four hour window.

We will give advance notice for Part 3 as soon as we have the date set. We appreciate your patience and understanding while we continue improving our network!

Update June 29, 3:45am PDT: Part 2 (software) of this network maintenance is now successfully completed. We will post a new status message when we’re ready go ahead with Part 3, the actual hardware move.

If you have any issues in the meantime, please contact support with the details, so we can assist you.

Previous updates available after the jump →


2:45 pm

WordPress 3.4.1 Security Release Now Available

Posted (June 27th, 2012 at 2:45 pm PST) by mir

We’re please to announce that the newest version of WordPress (3.4.1) is now available on the DreamHost panel for new installs and upgrades. Please be aware that this is a maintenance and security release and as such it is highly recommended for all customers.

You can find the release notes here:

http://wordpress.org/news/2012/06/wordpress-3-4-1/

With further details in the codex:

http://codex.wordpress.org/Version_3.4.1

Automatic upgrade (for customers that have them enabled via the DreamHost panel) will begin shortly. If you would like to see if you have automatic upgrades enabled (or want to run your own) you can find that on the panel in the same location where you installed WordPress:

https://panel.dreamhost.com/index.cgi?tree=goodies.installer&

Update June 29, 4:22pm PDT The automated upgrades are rolling out and so far things look good. If you are having any issues with your WordPress installation please be sure to check and see if you were set up for an automated upgrade in case that is the issue and contact support with any questions.


8:19 pm

Connection issues at one of our datacenters (RESOLVED)

Posted (June 19th, 2012 at 8:19 pm PST) by andrea

We are currently having connection issues at one of our Los Angeles data centers, which is also affecting our DNS servers. We’re working on resolving the situation and will update this post shortly. The DreamHost panel, webmail and dreamhost.com are unavailable. We are very sorry about the inconvenience.

Update June 19, 9:14pm PDT: We are confident that this incident is now resolved; all services are up and operational. We have been checking cases in the most recent incoming support and sites are loading, but if you’re still having issues with yours, please contact support with the details, so we can assist you.

Previous updates available after the jump →


2:38 pm

WordPress 3.4 now available for all new One-Click installs and upgrades via the DreamHost panel!

Posted (June 14th, 2012 at 2:38 pm PST) by mir

We’re excited to announce that we’ve implemented the most recent upgrade for WordPress for customers using our One-Click installs. Auto-upgrades, for customers who have them enabled, will begin shortly (so if you have that enabled you’re all set!). For those who have not selected that option you can process the upgrade from the panel where you installed WordPress:

https://panel.dreamhost.com/index.cgi?tree=goodies.installer&

While this is not a security update, it is chock-full of goodness and includes “significant improvements to theme customization, custom headers, Twitter embeds, and image captions”!

The full Release Notes can be found here: http://wordpress.org/news/2012/06/green/

Troubleshooting after an upgrade?

Check the WordPress Support Forums for a list of common issues and helpful tips:

http://wordpress.org/support/topic/troubleshooting-wordpress-34-master-list


5:15 pm

DNS Server Problems (RESOLVED)

Posted (June 13th, 2012 at 5:15 pm PST) by Ralph

We are experiencing some service interruption as a result of an issue with our DNS servers. This will cause sites to load slowly or hosting and email may not be working. We are working quickly to resolve this and will provide an update as soon as possible.

Update June 13, 7:55pm PDT: After waiting a bit for all issues to subside, we can confidently say this problem should now be resolved. All DNS servers involved are behaving normally, and are responding correctly to all tests. We have spot checked many sites, as well, and all perform normally. If you’re still seeing problems, please let our support staff know immediately! — Brian H

Previous updates available after the jump →


8:59 pm

Network Improvement Series Maintenance #4: Core Router Upgrade (Orange County) Saturday, June 9 –9pm PDT(4 hours) Part 2: Cut Over(REVERTED)

Posted (June 9th, 2012 at 8:59 pm PST) by Ralph

Part 2: Cut over to new hardware – Saturday, June 9th – 9pm PDT – 1am PDT ( 4 hour window). With the preparation and staging completed Thursday night, we will be able to cut over to the new hardware and use it in our live network environment. The process will involve moving individual subsets of our network one at a time. Service interruption will occur while the individual networks are moved, but will only affect customers on those specific networks for a short time (< 15 minutes). We do have multiple physical connections to switch so we expect to use the entire 4 hour window.

Update June 10th, 12:50am PDT: We have successfully reverted our changes and all sites affected by this change are working correctly again. We were able to work through some of the challenges that came up during the maintenance, but at this point we feel it is best to take the information gained and re-evaluate the best way to upgrade the core router at our data centers. We will provide you further information once we reschedule part 2 of Network Improvement Series #4.

Previous updates available after the jump →


11:06 am

Network issue in Garland datacenter (RESOLVED)

Posted (June 8th, 2012 at 11:06 am PST) by Brandon

We are currently experiencing network connectivity issues in our Garland data center. This is resulting in site downtime as well as panel unavailability. Our network engineers and system administration team are working to restore connectivity at this time. We will update this post with further information as it becomes available.

Update June 8th, 11:15am PDT:

This issue is now resolved.  If you are still noticing issues please contact support immediately and we will address your issue first.   We apologize for the service interruption and are happy to report that the fast reaction and hard work of our admin and NOC teams kept the downtime to under 20 minutes.


3:48 am

Network issue between 2 of our datacenters (RESOLVED)

Posted (June 8th, 2012 at 3:48 am PST) by patrickm

We apologize for any service interruption you may be experiencing. We are currently investigating an issue affecting traffic between 2 of our datacenters which is affecting connections from one server to another, most specifically in cases where a web server is in one datacenter and the MySQL server in another. This may show itself as failed MySQL connections on your websites.

Our Admins are actively working to resolve this issue and we will post when we have additional information or a resolution!

Update: This is now completely resolved! We were able to identify the issue as a configuration mismatch between the routers in two of our datacenters. Once identified, this was quickly resolved by our Network Engineering team.

As we continue our Network Improvement Series, we will be adding this to the checklist of items to be checked and double-checked to ensure this type of issue does not happen again.

If you are still experiencing any issues with your hosting services at this time, please feel free to Contact Support for further assistance!

Previous updates available after the jump →


6:58 pm

Network Improvement Series Maintenance #4: Core Router Upgrade (Orange County) Thursday, June 7 – 8pm PDT -10pm PDT(2 hours) Part 1: Staging (RESOLVED)

Posted (June 6th, 2012 at 6:58 pm PST) by Ralph

Our border router upgrades are now successfully complete, so it is time for us to move on to the core routers of our Orange County and Los Angeles based data centers. First up will be the Orange County data center at Irvine. We will be upgrading the core router in a two-part process.

1. Staging – Thursday June 7th, 8pm PDT- We will be installing the new router hardware and testing connectivity. Maintenance window will be 2 hours. This process will involve racking hardware, extending cables and testing the connectivity. We will not be routing any production connections through the new hardware. Service impact will be minimal, previous staging attempts have resulted in no downtime, but when making network changes we are always aware that service interruptions can occur.

Cut over to new hardware – Saturday June 9th, 9pm PDT- With the preparation and staging completed Thursday night, we will then be able to actually cut over to the new hardware and use it in our live network environment. This window will last 4 hours. The process will involve moving individual subsets of our network one at a time. Service interruption will occur while the individual networks are moved, but will only affect customers on those specific networks. This means that downtime for individual customers will be limited. We do have several physical connections to switch so we expect to use the entire 4 hour window.

We thank you for your continued patience as we progress further in our Network Improvement plans. We will update this post for the Thursday staging maintenance 10 minutes prior to starting.

Update June 7th, 8:45pm PDT: We have successfully completed the staging portion of this maintenance. The work is set up for the transfer of our production network to our upgraded core hardware on Saturday. Thank you as always for your patience and support.

Previous updates available after the jump →


 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
5:48 pm

Network Improvement Series Maintenance #3: Tuesday June 5 10pm PDT 4 hours-Part 3 Border Hardware Upgrade-Los Angeles Data Center(RESOLVED)

Posted (June 4th, 2012 at 5:48 pm PST) by Ralph

We will be continuing the Network Improvement Series Maintenance #3 Tuesday night, June 5th, at 10pm. We already have the new upgraded hardware installed and ready to go (that was completed during our Saturday maintenance window). We will now be routing traffic from our previous border router and on to the new, upgraded hardware. To minimize downtime during this maintenance, we’ll be moving our providers to the new router in steps. First, we’ll route all traffic through one provider and then move the remaining providers (without traffic) to the new router. Once that’s done, we will test the new router to make sure it’s handling traffic correctly. Then, we will switch all traffic to the transit provider that was moved to the new router. After that is finished we will then move the other providers (without active traffic) to the new router as well. Once that is complete we will have our new upgraded border installed and working which will allow us to proceed with Network Improvement Series Maintenance #4. Thank you for your continued patience and we will do our best to minimize the service impact with these changes.

Update June 6th, 2:30am PDT: Our upgraded border is now installed and working correctly in our Los Angeles-based datacenter! The final clean-up is finished and we have all sites working correctly and all network routes going through our upgraded border router. Please keep an eye out for the announcement of Network Improvement Series Maintenance #4, which will involve the core router at our Orange County datacenter.

Previous updates available after the jump →


7:00 pm

Network Improvement Servies Maintenance #3: Sat June 2 8pm PDT 4 hours-Part 2 Border Hardware Upgrade-Los Angeles Data Center (RESOLVED)

Posted (June 2nd, 2012 at 7:00 pm PST) by Brian@dreamhost

We will be attempting to reschedule Network Improvement Servies Maintenance #3 in multiple parts in order to ensure the successful upgrade. A summary of the 3 part process is as follows:

1: (completed during the Wednesday maintenance window) Moved network routes to route around the hardware we will be replacing to minimize service interruptions.
2: (Scheduled for Saturday June 2nd)
Connecting upgraded hardware and verifying it is working as expected.
3: (Scheduled for Tuesday June 5th)
Moving transits and going live on the upgraded hardware.

This part of the maintenance will occur this Saturday June 2nd at 8pm and will last for 4 hours. There is expected to be no service interruption during this maintenance as we will just be connecting the new hardware (on the backend) and verifying it is functioning as expected. No traffic will be routed to it from the outside. Although this is not expected to be customer facing, this is a network change and the nature of these changes can produce unexpected service interruption. This maintenance is laying the foundation for Tuesdays Part 3 maintenance where we will be finalizing the cut over to our new upgraded hardware. We appreciate your patience during these maintenance windows.

Update June 2nd 9:15pm PDT: The new hardware has been successfully connected and there were no service interruptions. Please keep in mind that Tuesday we will continue with Network Improvement Series Maintenance #3 where we will actually move the transits over to the new hardware.

Previous updates available after the jump →


12:24 am

Network Connectivity Issues (RESOLVED)

Posted (June 2nd, 2012 at 12:24 am PST) by patrickm

We apologize if you were affected by a brief service outage this evening, which occurred between 11:40pm PDT and 12:06am PDT which affected servers in some of our datacenters. We are currently investigating the cause of this outage and will be posting more details when we are positive we have found the root cause and implemented a solution!

We are spot-checking services at this time, but it appears that service has been restored for all customers. Please Contact Support if you are still experiencing any issues with your hosting and we will be happy to help!

Update Jun 2, 2012 1:51am PDT: We are happy to report that things have remained stable since the outage earlier and we are resolving this active post. Currently our investigation points to a hardware issue. We are continuing to monitor hardware and have verified redundant hot spares are in place and functional.

As always, if you are experiencing any issues with your hosting, please contact support for further assistance!


5:17 pm

Network Improvement Series Maintenance #3 Wednesday May 30 10pm PDT 4 hours – Upgrading Border Hardware at our Garland DC (RESOLVED)

Posted (May 25th, 2012 at 5:17 pm PST) by Ralph

We will be attempting to reschedule our previously postponed network maintenance to Wednesday May 30th. This maintenance is dependent on provider upgrades so we will have to set a “Go” or “No Go” on this particular maintenance. The decision to “Go or No Go” will be on Tuesday May 29th, however we are fully committed to perform this maintenance if our provider upgrades are complete.

We will be conducting planned maintenance on our Garland border router starting at Wednesday May 30th at 10pm Pacific Daylight Time (PDT). We expect the maintenance work to be completed by 11pm PDT, however for planning purposes we are specifying a 4 hour maintenance period through May 31st 2am PDT.
Here is the sequence of expected events during maintenance:
– Maintenance in our Garland data center will start between 10 – 10:30pm PDT, resulting in all web hosting services in that data center being offline;
– A new border router that has been set up and prepared for operation will be connected to the network and tested, with a potential duration of 45 minutes of service interruption, for the testing to be completed;
– Once the testing is completed and any issues are addressed to make the router fully operational, network services will be restored in Garland and all web hosting services will be back online; we expect this to occur by 11:30pm PDT, however it is possible that network services will not be restored until May 31st 1am PDT.
– In the event that the new router does not pass all tests by midnight PDT, we will restore services on the current network router hardware, so that in any scenario, all network services are restored by 12 midnight PDT.

Update Thursday May 31st 1:50am PDT: The first step, which involved rebalancing our traffic, was a success. We are however rolling back the hardware upgrade. We were not 100% comfortable with the progress that we made during the maintenance window. We will be rescheduling the hardware part of the upgrade as soon as we confident we can get through the issues we noticed today. All services are now restored. Thank you for your patience during this maintenance window.

Previous updates available after the jump →


9:50 pm

Network Improvement Series Maintenance #3 Wednesday May 23rd 10pm PDT 4 hours – Upgrading Border Hardware at our Garland DC (POSTPONED)

Posted (May 23rd, 2012 at 9:50 pm PST) by Ralph

We will be conducting planned maintenance on our Garland border router starting at Wednesday May 23 at 10pm Pacific Daylight Time (PDT). We expect the maintenance work to be completed by 11pm PDT, however for planning purposes we are specifying a 4 hour maintenance period through May 24th 2am PDT.
Here is the sequence of expected events during maintenance:

– Maintenance in our Garland data center will start between 10 – 10:30pm PDT, resulting in all web hosting services in that data center being offline;

– A new border router that has been set up and prepared for operation will be connected to the network and tested, with a potential duration of 45 minutes of service interruption, for the testing to be completed;

– Once the testing is completed and any issues are addressed to make the router fully operational, network services will be restored in Garland and all web hosting services will be back online; we expect this to occur by 11:30pm PDT, however it is possible that network services will not be restored until May 24th 1am PDT.

– In the event that the new router does not pass all tests by midnight PDT, we will restore services on the current network router hardware, so that in any scenario, all network services are restored by 12 midnight PDT.

Update May 23rd 10:30pm PDT: We have made a decision to postpone todays maintenance. During the final steps before the upgrade, our confidence in our transit connectivity was not at 100%. After exploring all possible options, we made the decision to postpone this upgrade, and keep your service running, as we assess the situation further. We expect this maintenance to be postponed till next Wednesday at the earliest. There was no service interruption due to any working being done tonight.

Update May 23rd 9:50pm PDT: We have been working hard preparing for the transition tonight and we are near ready to execute the hardware upgrade. In about 10 minutes we will begin work toward making the hardware switch. We will notify you with consistent status posts as we progress through the change. Thank you for your patience during this maintenance window.


11:17 am

Delay in sending email from web servers (RESOLVED)

Posted (May 22nd, 2012 at 11:17 am PST) by andrea

The mail relay machines for your web servers got backed up earlier today. We are working on processing that mail as fast as we can, but it will likely take another few hours to clear that mail queue. In the meantime, you will see delay in sending your php- or sendmail-based email or those created by your cronjobs. No mail is lost, we’re processing it all in the order they are generated on your server. We apologize for the delay and the inconvenience this is causing.

Update May 22nd 10:04pm PDT We’re happy to report the mail server has caught up now. Again, we are very sorry about the inconvenience this has caused. -Dan J

Previous updates available after the jump →


12:56 pm

Immediate Maintenance Required on Discussion List Servers ( RESOLVED )

Posted (May 16th, 2012 at 12:56 pm PST) by Oscar

Our system administration is performing maintenance on our discussion list servers. We expect all discussion list services to be down approximately one hour. We apologize for the inconvenience but it was deemed necessary as we’re setting up an improved (and more reliable) backup system for the lists.

UPDATE May 16th 3:18PM PDT The maintenance has completed. You are archives are now accessible. Again, we apologize for any inconvenience this has caused. If you notice any problems, please contact support for assistance. Previous updates available after the jump →


12:38 am

Networking issue at the Irvine datacenter (RESOLVED)

Posted (May 16th, 2012 at 12:38 am PST) by andrea

We’re experiencing connectivity issues at our Irvine datacenter. Our network engineers and system administration team are onsite working to restore connectivity. We will update this post as soon as we have more information.

Update May 16th 1:39am PDT Tonight’s connectivity issues appeared to be a result of a drop out experienced by one of our transit providers. They are dispatching an investigation team to help assist us in our diagnosing the exact issue that occurred. We are not expecting any further problems, however we will continue to monitor the situation, as well as assist our transit provider on finding further details on this issue. We’re resolving this case for now, but if you have further issues, please contact support for assistance.

Previous updates available after the jump →


12:19 am

Transit issue between LAX and Alchemy datacenters (RESOLVED)

Posted (May 12th, 2012 at 12:19 am PST) by Brian@dreamhost

Currently, at this time we’re seeing network degradation between our LAX and Alchemy datacenters due to maintenance by our transit provider. We are investigating this issue and will have more info as it progresses. This issue should only affect a small amount of customers with mysql databases that are in opposite datacenters, or users using webmail, on our homiemail-sub5 mail cluster — Brian H

Update May 12th 2:10am PDT This issue should now be taken care of at this time. We were able to configure a different link with another provider, and restore service. We apologize for any issues this may have caused.

Previous updates available after the jump →


1:30 pm

Network Improvement Series Maintenance #2 Monday May 14th 10pm PDT 4 hours – Upgrading Border Hardware at our Irvine DC (RESOLVED)

Posted (May 10th, 2012 at 1:30 pm PST) by Ralph

This is the second step in our DreamHost Network Improvement Series.

On Monday, we will be upgrading the Border Hardware at our Irvine Data Center. This is an important step in our plans to provide you the best in network performance and stability. We will be giving ourselves a 4-hour window to complete the installation and configuration of the new border, however, we expect that any actual outage to be minimal. Our team will stay on hand to make sure the transition is as smooth as it can be. Any sites or email hosted in this data center will have no network path to them, so your site will be down and your email should queue up on servers sending messages to you. We thank you for your patience as we continue to work to improve your DreamHost experience.

Update May 14th 10:08pm PDT: We started our maintenance at 10pm PDT, and experienced less than a minute of downtime. Our customer test domains are loading and are stable, and we are confident that this case can now considered to be resolved. Our Irvine datacenter now has new border hardware.

If you’re having issues with your sites, it’s likely unrelated, so please contact support with the details, so we can assist you.

Previous updates available after the jump →


 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
9:00 pm

Network Improvement Series Maintenance #1 Wednesday May 9th 10pm PDT 4 hours – Upgrading Border Hardware at our LAX Data Center (RESOLVED)

Posted (May 9th, 2012 at 9:00 pm PST) by Ralph

This is the first step in our DreamHost Network Improvement Series.

On Wednesday we will be upgrading the Border Hardware at our LAX Data Center. This is an important step in our plans to provide you the best in network performance and stability. We will be giving ourselves a 4 hour window to complete the installation and configuration of the new border, however, we expect that any actual outage to be minimal. Our team will stay on hand to make sure the transition is as smooth as it can be. Any sites or email hosted in this data center will have no network path to them, your site will be down and your email should queue up on servers sending messages to you. We thank you for your patience as we continue to work to improve your DreamHost experience.

Update May 10th 1:10am PDT: We are considering this maintenance resolved. Thank you for your patience as we worked through this first installment of our Network Improvement Series. We are excited about these changes and what they mean to our service and what they’ll mean to you, our customers, as well.

Previous updates available after the jump →


7:00 pm

Tuesday May 15th Network Maintenance – fiber ring between our two Los Angeles based data centers – 10pm PDT – 2 hours

Posted (May 2nd, 2012 at 7:00 pm PST) by Ralph

Previously this was to be done on Wednesday May 2nd, but we moved it to the 15th.

On Tuesday, May 15th, we will be performing fiber maintenance that will confirm our transfer capacity between our Los Angeles-based data centers. The fiber ring is being tested to minimize errors and to ensure maximum throughput. We are giving ourselves a two-hour window starting at 10pm PDT Tuesday May 15th. Service affected will be web and email. Websites that have their hosting server in one data center and their MySQL databases in the other will notice database connection errors. MySQL and web server in different data centers is not our typical setup, so this should only affect a small subset of customers. This maintenance should not take the full two hours to complete. We will be using the maintenance window to monitor the changes and make sure no complications arise. This upgrade is extremely important to our reliability focus and efforts.

Update May 16th 12:17pm PDT We have to postpone the maintenance while our fiber ring provider works out the problem on their that caused the unexpected issues. We’ll put up a new status post as soon as we know the new schedule.

The DreamHost panel is up and running again. Sorry about the inconvenience!

Previous updates available after the jump →


12:53 pm

Networking Issue at Irvine DataCenter (RESOLVED)

Posted (April 28th, 2012 at 12:53 pm PST) by danwy

We have noticed connection issues in our Irvine datacenter. This is resulting in site and database downtime. Our network engineers and system administration team are onsite working to restore connectivity. We will update this post as soon as we have more information.

Update April 28th 1:20pm PDT We apologize for the continued down time but our network engineers have reported progress and we believe we’ll be fully operational again shortly.

Update April 28th 1:42pm PDT We would like to report that all of the sites we have tested that were experiencing issues are back up and loading for us. We’re continuing to monitor things closely but it appears that this matter has been resolved. We will provide more details when we have been able to assimilate all of the feedback from our admin and networking team members.

Update May 2nd 2:20pm PDT While this issue has been fixed since our last update, we kept this post unresolved until we were able to get more information about the specifics of it. This issue was caused by another DoS attack. During the process of blocking the external origin of the attack, the core routers became unreachable and therefore blocked traffic to Irvine. We had to reboot the routers to solve the problem and return to a stable state. We have already planned on replacing these routers with much more powerful ones in the next couple of weeks to help avoid future issues like this.


6:52 pm

Wednesday May 2nd Network Maintenance – fiber ring between our two Los Angeles based data centers – 10pm PDT – 2 hours (MOVED TO MONDAY – NEW POST)

Posted (April 27th, 2012 at 6:52 pm PST) by Ralph

On Wednesday May 2nd we will be performing fiber maintenance that will confirm our transfer capacity between our Los Angeles based data centers.  The fiber ring is being tested to minimize errors and to ensure maximum throughput.  We are giving ourselves a 2 hour window starting at 10pm PDT Wednesday May 2nd.  Service affected will be strictly web content. Websites that have their hosting server in one data center and their MySQL databases in the other will notice database connection errors. MySQL and web server in different data centers is not our typical setup so this should only affect a small subset of customers.  This maintenance should not take the full 2 hours to complete. We will be using the maintenance window to monitor the changes and make sure no complications arise. This upgrade is extremely important to our reliability focus and efforts.

Update May 2nd 7pm PDT: This maintenance window has been cancelled and will be moved to Monday May 7th at 10pm. A new status post will be created to keep track of the outage.


7:14 pm

Wednesday April 25th Network Maintenance Irvine Data Center 10pm PDT – 2 hour window (RESOLVED)

Posted (April 25th, 2012 at 7:14 pm PST) by Ralph

In our continued efforts to upgrade our network, we will be scheduling maintenance on a piece of our border network equipment on April 25th. The action being performed is a simple software upgrade which will improve resource utilization on the new Juniper router we installed. We anticipate the downtime itself to last no longer than 10 minutes; however, we will be on high alert for the 2 hours preceeding the window as well as during and after the upgrade to make sure there are no complications. We sincerely appreciate your patience and understanding as we strive to improve our service for you.

Update April 25th 10:54pm PDT We’re happy to report that this upgrade is done, and happened without any issues! All service should be working as expected. If you’re seeing problems with your domains, please contact support immediately. — Brian H

Update April 25th 10:21pm PDT First part of upgrade is complete. We are starting the final upgrade now, then one more reboot and we will be upgraded. We will continue to monitor after upgrade is complete.

Update April 25th 10:12pm PDT The reboot of the network equipment is happening right now.

Update April 25th 10:00pm PDT Maintenance has started. You will notice an interruption of service as we upgrade the hardware. We will keep you posted as we get close to completing this upgrade.

Update April 25th 7:15pm PDT Maintenance is scheduled to start tonight at 10pm. Please keep an eye on this post for updates once the maintenance starts.


2:02 pm

WordPress 3.3.2 update available for one-click installs.

Posted (April 20th, 2012 at 2:02 pm PST) by Brandon

WordPress 3.3.2 is now available for new installs and upgrades via our One-Click Installer system.

As this is a security release, it’s a recommended upgrade for all users who run WordPress.

Auto-upgrades have been queued for all customers who have them enabled, but will take some time to process.

You can check if you have automatic upgrades enabled (and upgrade manually if you prefer not to wait) from the One-Click Installer section of the DreamHost Panel:

* Log into the DreamHost Panel, and go to the One-Click Installer Section https://panel.dreamhost.com/?tree=goodies.installer
* Click on “Managed Installed Applications”
* Click on your domain, and a list of your installs will show
* Under the “Upgrade Action” column, you can view and modify the status of auto-upgrades for each install
* Click on the “Upgrade to 3.3.2″ button to the right of the install if you wish to upgrade right away

If you installed WordPress yourself, rather than using our One-Click Installer system, you can upgrade by logging into your WordPress Dashboard and clicking “Please update now” from the alert at the top of the screen.

You can see full release notes at WordPress.org:
http://wordpress.org/news/2012/04/wordpress-3-3-2/


3:29 pm

Network Issue at Irvine DataCenter (RESOLVED)

Posted (April 17th, 2012 at 3:29 pm PST) by Dan J

We are noticing connection issues to our Irvine datacenter. Our network engineers and system administration team are working to restore connectivity.  We will update this post once we have more information.

Update Apr 17, 4:07pm PDT: The issue is now resolved, and lasted for about 20 minutes. Our network engineers and system administrators worked quickly to recover the network, and are continuing to work on any possible residual issues. We will make sure to keep you updated, as well as provide you with details regarding the root cause as soon as we have them. We sincerely appreciate your patience while we continue to get this resolved. -Seohee

Update Apr 17, 5:45pm PDT: We sincerely apologize for the network issues experienced at our Irvine datacenter. This particular incident was a routing table issue and was not associated with the previous outages we experienced at Irvine. This outage was smaller in scope and took down services for 20 minutes. The configuration issue that caused this took down 1 of 4 subnets used in Irvine. The fix is in place now and we will avoid any further problems due to this particular issue. We are continuing to work through significant network improvements and have made this our top priority.


6:03 pm

Emergency Network Maintenance 4/12 – 6 hour window (RESOLVED)

Posted (April 12th, 2012 at 6:03 pm PST) by Ralph

Tonight we will be doing some hardware changes to boost our network at or Irvine data center. This decision is on the heels of multiple DDoS attacks that have happened today which caused service interruptions. We will be bolstering our border routers by having new hardware installed which will manage our connections to the peering partners. This will allow our BGP sessions to be stable through any possible DDoS that can affect our main border router. As this maintenance starts we will provide you with updates as we make progress.

Update Apr 13, 1:45am PDT: We are resolving this post. Everything is working as expected now with some minor adjustments to be made tomorrow. We will keep you updated if any further major changes are needed in the coming days. Thank you for your continued patience and if you have any downtime issues please contact support.

Previous updates available after the jump →


3:46 pm

Network Issue at Irvine DataCenter (RESOLVED)

Posted (April 12th, 2012 at 3:46 pm PST) by Oscar

We are noticing intermittent unavailability to our Irvine datacenter again. This is due to another Distributed Denial of Service (DDoS) attack. Our network engineers and system administration team are working on mitigating this attack.  We will update this post once we have more information.

Update Apr 13, 1:54am PDT: Our maintenance window has concluded successfully and all services are restored. If we have any further plans we will let you know in a new status post. Thank you for your patience.

Update Apr 12, 6:10pm PDT: We’re proceeding with an upgrade to our infrastructure which should not only alleviate the current issue but also preclude us from suffering similar issues in the future. Please refer to our new status post with details on that action. We will update this post again once that has completed and we have had a chance to access the results. — Brian H

Previous updates available after the jump →


9:22 am

Network Issue at Irvine DataCenter (RESOLVED)

Posted (April 12th, 2012 at 9:22 am PST) by brians

We are currently experiencing issues with our Irvine datacenter. We have our network team looking into it and are working to get service restored as soon as possible. As soon as we have further information we will be updating this post.

Update Apr 12, 3:25pm PDT: We are confident that this incident is resolved now.  Distributed Denial of Service (DDoS) attacks triggered a instability of BGP routes to our peering partner.  Now that we understand the source of the outage we have installed a DDoS mitigation appliance to address future DDoS attacks.  We will also be announcing some further maintenance to our routing infrastructure which will compliment the current hardware and assume the role of managing our BGP routes.  The addition of the DDoS solution will help prevent near term issues, but our commitment is to upgrade our networking infrastructure to provide you the stability you expect.  Please keep an eye on this status post for future maintenance announcement.  – Oscar Previous updates available after the jump →


 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
6:23 pm

Minor switch issue (RESOLVED)

Posted (April 9th, 2012 at 6:23 pm PST) by Ralph

We are looking into a switch issue that is affecting two racks at one of our data centers. The list of machines is listed below.

andromeda, antila, apus, aquarius, aquila, ara, auriga, bootes, bullhead,
caelum, camelopardalis, cancer, canis, capricornus, carina, cassiopeia,
centaurus, cepheus, chamaeleon, circinus, columba, crux, cygnus, delphinus,
dogfish, dorado, draco, equuleus, eridanus, fornax, gemini, grus, horologium,
indus, lacerta, lepus, megamouth, sandtiger

We are addressing the issue quickly and will provide an update as soon as we have more information.

Update Apr 9, 7:20pm PDT: This issue is resolved now – switching ports resolved the issue. We will continue to monitor the switch for any further signs of instability and, in addition to that have begun the process of replacing the hardware involved to preclude any future incidents.

Update Apr 9, 7:10pm PDT: We’ve migrated the uplink on the affected switch to another port and some connectivity has been restored. However, we are still seeing minor packet loss to various IPs on that switch so there still remains an outstanding service degradation issue. Our Network and Systems admins are still collaborating on the issue and we’ll post more information as it becomes available.

Update Apr 9, 6:49pm PDT: Switch is up and we’ve confirmed management level connectivity however it is stll unable to get to the outside world. We are continuing to troubleshoot the switch config as well as its uplinks to the cores

Update Apr 9, 6:43pm PDT: We have rebooted the switch and are still investigating this issue. We will update this post regularly till resolution.


8:15 am

Irvine Datacenter Connection Issues (RESOLVED)

Posted (April 9th, 2012 at 8:15 am PST) by Justin@DH

We are currently experiencing issues with our Irvine datacenter caused by a DDoS attack. We have our network team looking into it and are working to get service restored as soon as possible. As soon as we have further information we will be updating this post.

Update Apr 9, 9:21am PDT:We are happy to report that all services were restored around 8:40am. We have been spot-checking various services to confirm that the network was fully functional. Our network engineers will continue to actively monitor the stability of our network. We apologize for any inconvenience this has caused and appreciate your continued patience. If you happen to notice any issues still, please contact support and we would be happy to assist you. – Justin

Previous updates available after the jump →


3:43 am

Intermittent network issues affecting some customers (RESOLVED)

Posted (April 7th, 2012 at 3:43 am PST) by patrickm

Due to issues caused by maintenance being performed by one of our transit partners, an upstream link we have, become unstable. This forced us to have to shut down the link for the time being. We will re-enable it once the maintenance is complete.

Update Apr 7, 6:45am PDT: The maintenance has ended, the connection was re-enabled and been running problem free for 1.5 hours. -JJ G

Services that were disrupted by this outage are now back online. If you are experiencing any issues they are likely unrelated. Please contact support and we will be happy to help! -Patrick


4:02 pm

Improvements to outgoing spam prevention policy

Posted (April 4th, 2012 at 4:02 pm PST) by Brian@dreamhost

In the ongoing battle against spam it has become necessary that we prevent outgoing mail from spoofing other major email providers including: hotmail.com, live.com, gmail.com, yahoo.com, aol.com, earthlink.net.

Forging / spoofing email sender addresses to an address that a DreamHost mail server does not have a logical right to send as creates problems when dealing with spam and blacklists, particularly with major providers that are receiving spam from their own domains. This change is just one of many measures we are working on will help strengthen the reputation of DreamHost’s mail servers for all of our customers and help prevent future blacklistings.

If you are currently sending as one of the above domains via your DreamHost email user you will need to update your mail client settings to send as your proper domain email address. If you do wish to send as one of the prohibited domains (such as Gmail) you will need to configure that email account through the proper email server on your computer/device.

This may also affect those that have scripts that send email as a form-input email address, or a configured admin address. These forms/scripts/CMS settings need to be changed to send as a proper address at your domain.

If you’re affected by any of this, not to fear, we’re here to help, so just drop us a line here. We’re making these changes to address spam being sent from our network. If you’re sending legitimate mail, then we’ll make sure you’re still able to do that!

Update Apr 5th, 9:25am PDT: Just to clarify, we didn’t roll back this entire change, but just the aspects of it that were causing mail to incorrectly bounce. Customers still will not be able to send mail using outside domains through our system, including the ones above (hotmail.com, live.com, gmail.com, yahoo.com, aol.com, earthlink.net). Sorry for the confusion. -dan w

Previous updates available after the jump →


5:25 pm

Planned maintenance on ‘ash’ web server Tonight Monday, April 2nd @10pm (PDT) (RESOLVED)

Posted (April 2nd, 2012 at 5:25 pm PST) by Dan J

Tonight Monday, April 2nd @ 10pm (PDT), the ‘ash’ servers will be moved to a new chassis. During this time, any sites hosted on this machine will be unavailable. This will not affect email or MySQL services, only web services for domains hosted on this server. To check if you are on this server, please check your user’s home machine in the DreamHost panel.

We expect the maintenance window to last less than 15 minutes and we will be updating this status post with information as the upgrades progress. We’re sorry for any inconvenience this causes you.

Please note, this post is only regarding the above upcoming maintenance. If you are having any issues with your hosting service outside the maintenance window, please see the other open posts and submit a support ticket through DreamHost panel with your details. We will be happy to assist you.

Update Apr 2, 10:24pm PST: The move has now completed. Everything went smoothly, total downtime about 10min. If you have any more trouble then please contact support through the Dreamhost panel. -Dan J

Previous updates available after the jump →


11:49 am

Connectivity issues with one of our datacenters ( RESOLVED )

Posted (March 23rd, 2012 at 11:49 am PST) by Brandon

We are currently experiencing some connectivity issues to one of our datacenters(more specifically, there is a small subset of our network which is unresponsive, fortunately the majority of our network and customer base is not impacted). Our networking, system engineers and datacenter operations teams have been deployed and are investigating. Sorry about the inconvenience this is causing. Please check back here often, as we will be posting updates as we receive them.

This issue is still only affecting a small subset of our customers, but will be affecting mail, webmail and MySQL.  Connectivity issues are specific to our internal servers being able to communicate with each other, and this doesn’t indicate an inability for the servers to communicate with machines outside of our own network. As of 12:50 pm we are currently in the process of re-routing traffic around the affected networks in order to restore communication. We don’t have a concrete time to completion just yet, but we are not anticipating much longer before there is noticeable improvement. We will continue to update this post with new information as it becomes available.

Update Mar 23rd, 5:01pm PST: We are happy to report that all services were restored at 3:30PM. We have been spot-checking various services to definitively confirm that the network was fully functional. Our network engineers will continue to actively monitor the stability of our network. We apologize for any inconvenience this has caused and appreciate your continued patience. If you happen to notice any problems, please contact support immediately. -Brian S

Previous updates available after the jump →


8:01 pm

Web VPS host server ‘homie-vserver194′ moving to new hardware and restoring from backups (RESOLVED)

Posted (March 20th, 2012 at 8:01 pm PST) by Seohee

Due to issues with the hard drive on the web VPS host machine ‘homie-vserver194′, the /home partition has become RO (read-only). We are preparing the new hardware now for the move, and will be restoring all data from our backup server to the new hardware once the move is in progress. During this process, the data on the backup server will be mounted directly over NFS to allow all services to remain online for the duration of the restoration.

We apologize for the inconvenience this is causing. We will update this post as soon as the new hardware is setup and the move has begun. If you have any questions or concerns, please contact Support under Support > Contact Support, in your panel.

Please note, this is regarding the web VPS host machine ‘homie-vserver194′ and no other services (email, MySQL) or servers are affected. To check if you are on a VPS guest from this host, please go to the Users > Manage Users page in your Web Panel and check the “machine” for your user. The affected VPS guests can be found in the first comment below this post.

Update Mar 29th, 2:35pm PDT: This restore is now complete. If you are experiencing any issues, or are missing any data from your VPS, please contact support for assistance! – Oscar

Previous updates available after the jump →


4:51 pm

Network issues in Irvine datacenter causing delays to server updates (RESOLVED)

Posted (March 19th, 2012 at 4:51 pm PST) by Dan J

Our Irvine datacenter is currently having packet loss issues with our backend network. This is preventing some updates from the Dreamhost panel to servers in the Irvine datacenter. This includes things like FTP user creation, password changes, changes to website configurations done through the Dreamhost panel.  This is also causing some problems with dedicated servers ability to reboot properly so we recommend that you do not reboot your dedicated servers at this time. The existing websites should still be online as the front end network is working properly. Our admin team is working as fast as they can to bring the backend network online again. All updates made in the Dreamhost panel should all go out once the backend network access has been restored. We apologize for the delays.

Update Mar 19th, 6:05pm PDT: The issue has been resolved now. If your site appears to be having issues still, please contact support. – Dan J


5:34 am

Web VPS host server ‘homie-vserver135′ moving to new hardware and restoring from backups (RESOLVED)

Posted (March 19th, 2012 at 5:34 am PST) by chalise

Due to issues with the RAID controller on the web VPS host machine ‘homie-vserver135′, we are in the process of moving the server on to new hardware and will be restoring all data from our backup server to the new hardware. During this process, the data on the backup server will be mounted directly over NFS to allow all services to remain online for the duration of the restoration.

We apologize for the inconvenience this is causing. We will keep this post updated with as much detail as we can provide. If you have any questions or concerns, please contact Support under Support > Contact Support, in your panel.

Please note, this is regarding the web VPS host machine ‘homie-vserver135′ and no other services (email, MySQL) or servers are affected. To check if you are on a VPS guest from this host, please go to the Users > Manage Users page in your Web Panel and check the “machine” for your user. The affected VPS guests can be found in the first comment below this post.

Update Mar 24th, 4:05pm PDT: This restore process is now complete. If you are experiencing any issues with your VPS, please Contact Support for assistance! – Oscar Previous updates available after the jump →


4:22 pm

Connectivity problems on our backend (RESOLVED)

Posted (March 18th, 2012 at 4:22 pm PST) by danwy

We are currently experiencing what seems to be some networking issues on our backend infrastructure. This is causing mail login issues for a small portion of our customers, as well as causing the webmail interface to become unavailable. We already have a team of admins on the job to diagnose the issue and resolve it. We don’t have many details at the moment, but we will update this post as soon as we have any additional information about this incident.

Update Mar 18th, 5:20pm PDT: We have temporarily resolved this issue by routing network connections around the defective switch. We’re still not entirely sure what caused the switch to fail, but our Admin team will continue to investigate it further. We’re still in the process of replacing this defective hardware, but we are doing it behind the scenes so that it does not affect any customer services. We will fully test the new switch hardware before we route traffic through it again. You shouldn’t notice any further downtime related to this incident. If you experience any further problems with email or webmail, please Contact Support.

Previous updates available after the jump →


 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
4:15 pm

Web VPS host server ‘homie-vserver159′ offline due to unscheduled file-system check (RESOLVED)

Posted (March 17th, 2012 at 4:15 pm PST) by Oscar

We apologize for the short notice, but the web VPS host machine, ‘homie-vserver159’ has just been taken offline to complete a file-system integrity check (fsck). The server was rebooted to resolve an issue with the file-system switching to read-only and this check needs to be completed to verify the integrity of the RAID array anytime this situation arises.

We apologize for the inconvenience this is causing. We will keep this post updated with as much detail as we can provide. If you have any questions or concerns, please contact Support under Support > Contact Support, in your panel.

Please note, this is regarding the web VPS host machine ‘homie-vserver159′ and no other services (email, MySQL) or servers are affected. To check if you are on a VPS guest from this host, please go to the Users > Manage Users page in your Web Panel and check the “machine” for your user. The affected VPS guests can be found in the first comment below this post.

Update Mar 17th, 9:34pm PDT: OK, the filesystem check has finshed and the server is back online now. If you notice any problem with your site then please  Contact Support. Thanks! -Dan J

Previous updates available after the jump →


1:26 pm

Web VPS host server ‘homie-vserver20′ moving to new hardware and restoring from backups (RESOLVED)

Posted (March 16th, 2012 at 1:26 pm PST) by brians

Due to issues with the RAID controller on the web VPS host machine ‘homie-vserver20′, we are in the process of moving the server on to new hardware and will be restoring all data from our backup server to the new hardware. During this process, the data on the backup server will be mounted directly over NFS to allow all services to remain online for the duration of the restoration.

We apologize for the inconvenience this is causing. We will keep this post updated with as much detail as we can provide. If you have any questions or concerns, please contact Support under Support > Contact Support, in your panel.

Please note, this is regarding the web VPS host machine ‘homie-vserver20′ and no other services (email, MySQL) or servers are affected. To check if you are on a VPS guest from this host, please go to the Users > Manage Users page in your Web Panel and check the “machine” for your user. The affected VPS guests can be found in the first comment below this post.

Our admin team is currently preparing the hardware move and restore process. Unfortunately, all VPS guests on this machine are temporarily offline while this occurs. We will post another update shortly with more news. Thanks for your patience! – Brian S

Update Mar 20th, 12:45pm PDT: The issue with the network interface has been corrected, and the  restoration has completed! All VPS guests are online, but if you’re experiencing any issues with your sites, please contact support . – Brandon

Previous updates available after the jump →


8:45 am

Web VPS host server homie-vserver211 moved to new hardware and restoring from backups (RESOLVED)

Posted (March 15th, 2012 at 8:45 am PST) by Justin@DH

Due to issues with the RAID controller on the web VPS host machine ‘homie-vserver211′. The admins are seeing if the server can be repaired via a file-system check, or if a move to new hardware , we are in the process of moving the server on to new hardware and will be restoring all data from our backup server to the new hardware. During this process, the data on the backup server will be mounted directly over NFS to allow all services to remain online for the duration of the restoration.

We apologize for the inconvenience this is causing. We will keep this post updated with as much detail as we can provide. If you have any questions or concerns, please contact Support under Support > Contact Support, in your panel.

Please note, this is regarding the web VPS host machine ‘homie-vserver234′ and no other services (email, MySQL) or servers are affected. To check if you are on a VPS guest from this host, please go to the Users > Manage Users page in your Web Panel and check the “machine” for your user. The affected VPS guests can be found in the first comment below this post. Our admin team is currently preparing the hardware move and restore process. Unfortunately, all VPS guests on this machine are temporarily offline while this occurs. We will post another update shortly with more news. Thanks for your patience!

Update Mar 15th, 11:00pm PDT: We are happy to announce that the restore has completed and the server is now running successfully from its new hardware. If you have any questions, concerns, or issues, please contact support so we can help! – Chalise

Previous updates available after the jump →


11:59 pm

Web VPS host server ‘homie-vserver211′ offline for file system check (RESOLVED)

Posted (March 13th, 2012 at 11:59 pm PST) by chalise

The web VPS host server ‘homie-vserver211′ is offline for an unscheduled file-system integrity check. This is required due to issues with the RAID on the server switching to read-only mode following a reboot of the server. We will provide frequent updates below as the check completes. We apologize for the inconvenience that this situation causes!

We apologize for the inconvenience caused by this unplanned outage. We will keep this post updated with as much detail as we can provide. If you have any questions or concerns, please contact Support under Support > Contact Support, in your panel.

Please see the first comment below to see a list of affected VPS guest servers.

Update Mar 14th, 2:45am PDT: The file check has completed! Thank you again for your patience while this ran. – Chalise

Previous updates available after the jump →


4:52 pm

Web VPS host server ‘homie-vserver234′ moving to new hardware and restoring from backups (RESOLVED)

Posted (March 13th, 2012 at 4:52 pm PST) by Dan J

Due to issues with the RAID controller on the web VPS host machine ‘homie-vserver234′, we are in the process of  moving the server on to new hardware and will be restoring all data from our backup server to the new hardware. During this process, the data on the backup server will be mounted directly over NFS to allow all services to remain online for the duration of the restoration.

We apologize for the inconvenience this is causing. We will keep this post updated with as much detail as we can provide. If you have any questions or concerns, please contact Support under Support > Contact Support, in your panel.

Please note, this is regarding the web VPS host machine ‘homie-vserver234′ and no other services (email, MySQL) or servers are affected. To check if you are on a VPS guest from this host, please go to the Users > Manage Users page in your Web Panel and check the “machine” for your user. The affected VPS guests can be found in the first comment below this post.

Our admin team is currently preparing the hardware move and restore process. Unfortunately, all VPS guests on this machine are temporarily offline while this occurs. We will post another update shortly with more news. Thanks for your patience!

Update Mar 17th, 4:50pm PDT: The restore process is now complete. All guests are up at this time. If you run into any problems with a VPS on this host machine or are missing any data then please contact support. -Dan J

Previous updates available after the jump →


6:46 am

Web VPS host server ‘homie-vserver186′ moving to new hardware and restoring from backups (RESOLVED)

Posted (March 13th, 2012 at 6:46 am PST) by chalise

Due to issues with the RAID controller on the web VPS host machine ‘homie-vserver186′, we are in the process of  moving the server on to new hardware and will be restoring all data from our backup server to the new hardware. During this process, the data on the backup server will be mounted directly over NFS to allow all services to remain online for the duration of the restoration.

We apologize for the inconvenience this is causing. We will keep this post updated with as much detail as we can provide. If you have any questions or concerns, please contact Support under Support > Contact Support, in your panel.

Please note, this is regarding the web VPS host machine ‘homie-vserver186′ and no other services (email, MySQL) or servers are affected. To check if you are on a VPS guest from this host, please go to the Users > Manage Users page in your Web Panel and check the “machine” for your user. The affected VPS guests can be found in the first comment below this post.

Our admin team is currently preparing the hardware move and restore process. Unfortunately, all VPS guests on this machine are temporarily offline while this occurs. We will post another update shortly with more news. Thanks for your patience!

Update Mar 20th, 12:45pm PDT: The restoration to the new hardware has completed. All VPS guests are online, but if you’re experiencing any issues with your sites, please contact support . – Brandon

Previous updates available after the jump →


3:04 pm

Planned maintenance on ‘stampeders’ and ‘michaelkelly’ web servers on Thursday, March 15th @10pm (PDT)

Posted (March 12th, 2012 at 3:04 pm PST) by danwy

This Thursday, March 15th @ 10pm (PDT), the ‘stampeders’ and ‘michaelkelly’ servers will be proactively upgraded to new hardware to help prevent any future issues on them. During this time, any sites hosted on these machines will be unavailable. This will not affect email or MySQL services, only web services for domains hosted on these servers. To check if you are on this server, please check your user’s home machine in the DreamHost panel.

We expect the maintenance window to last about an hour and we will be updating this status post with information as the upgrades progress. We’re sorry for any inconvenience this causes you.

Please note, this post is only regarding the above upcoming maintenance. If you are having any issues with your hosting service outside the maintenance window, please see the other open posts and submit a support ticket through DreamHost panel with your details. We will be happy to assist you. -Dan

Update Mar 18th 7:30am PDT – This maintenance was completed. No further issues have been detected.


12:01 pm

Scheduled network maintenance in one of our datacenters on Tuesday, March 13th at 10pm (RESOLVED)

Posted (March 12th, 2012 at 12:01 pm PST) by andrea

We are scheduling an overall network maintenance for one of our datacenters for Tuesday between 10pm and 12am PDT. This will only affect a small subset of our customers. We estimate the actual service interruption for web, mysql, email and the DreamHost panel to be no more than 15 minutes. We apologize for any inconvenience this may cause.

Please note, this post is only regarding the upcoming maintenance. If you are having any issues with your hosting service outside the maintenance window, please see the other open posts and submit a support ticket through DreamHost panel with your details. We will be happy to assist you. -Andrea

Update Mar 13th, 11:45pm PDT: Network maintenance has completed! Thank you, once again, for your patience! If you notice any issues with your sites or any of your services, please contact support as it is not likely related to this. – Chalise

Previous updates available after the jump →


7:37 am

Web VPS host homie-vserver127 is running file-system check (RESOLVED)

Posted (March 11th, 2012 at 7:37 am PST) by Justin@DH

The web VPS host ‘homie-vserver127 is currently getting a file-system integrity check. This is required due to complications with the RAID on the server which caused switching to read-only mode following a reboot. During this, the host can’t be online, but we’re working on mounting your data from backups, so we can still serve your sites in the meantime. We apologize for the inconvenience that this situation causes! Please contact support if you have any questions!

Please note, this is regarding the VPS host machine ‘homie-vserver142’ and no other services (email, MySQL) or machines are affected. The affected VPS guests on this host can be found in the first comment below this post. To check if you’re on this server, please go to the DreamHost Panel to check your VPS guest number. We will continue to update this post as information is made available. Please check back for updates.

Update Mar 11th, 9:00am PDT: The file system check has completed and the server is running again. We will be considering if any further action is needed and will be verifying that the backups are updated. – Justin

Previous updates available after the jump →


12:07 pm

Shared web server ‘dubhe’ moved to new hardware and restored from backups (RESOLVED)

Posted (March 10th, 2012 at 12:07 pm PST) by Oscar

Due to issues with the RAID controller on the shared web server ‘dubhe’, we have moved the server on to new hardware and are restoring all data from our backup server to the new hardware. During this process, the data on the backup server will be mounted directly over NFS to allow all services to remain online for the duration of the restoration.

We apologize for the inconvenience caused by this unplanned outage. We will keep this post updated with as much detail as we can provide. If you have any questions or concerns, please contact Support under Support > Contact Support, in your panel.

Please note, this is regarding the shared web server ‘dubhe’ and no other services (email, MySQL) or servers are affected. To check if you are on this server, please check your user’s home machine in the DreamHost panel.

Update Mar 12th, 4:20pm PDT: This restore is now complete, and all sites are up and resolving. If you notice any issues with your sites or are missing date, please Contact Support. -Brandon

Previous updates available after the jump →


 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
7:27 am

Shared web server ‘kingston’ moved to new hardware and restoring from backups (RESOLVED)

Posted (March 10th, 2012 at 7:27 am PST) by Oscar

Due to issues with the RAID controller on the shared web server ‘kingston’, we have moved the server on to new hardware and are restoring all data from our backup server to the new hardware. During this process, the data on the backup server will be mounted directly over NFS to allow all services to remain online for the duration of the restoration.

We apologize for the inconvenience caused by this unplanned outage. We will keep this post updated with as much detail as we can provide. If you have any questions or concerns, please contact Support under Support > Contact Support, in your panel.

Please note, this is regarding the shared web server ‘kingston’ and no other services (email, MySQL) or servers are affected. To check if you are on this server, please check your user’s home machine in the DreamHost panel.

Update Mar 18th, 1:15am PDT: This restore has now been completed and the issue has been resolved. However, please contact support if you are experiencing any issues with your hosting on this server! –JJ G Previous updates available after the jump →


5:21 am

Customer Web Panel is offline (RESOLVED)

Posted (March 10th, 2012 at 5:21 am PST) by patrickm

We are very sorry for the inconvenience that this may be causing. The Web Panel is currently offline and our Admins are investigating the issue. While the Web Panel is offline, you will not be able to make changes to your hosting settings. We are working to get this back online and will post updates to the situation below.

This is affecting just the Web Panel and no hosting services whatsoever. All other services are online, so if you are experiencing any issues with your hosting it is unrelated to this issue and you should contact support for assistance when the Web Panel is back online.

Thank you for your patience and check back with this post for updates to this situation.

Update Mar 10th, 6:07am PST: This issue is now resolved! Our Admins were able to get the servers for the Web Panel working correctly again and everything is back to normal. If you are experiencing any issues, please contact support! -Patrick


3:43 pm

Security improvements and password change reminder

Posted (March 9th, 2012 at 3:43 pm PST) by kathy

At DreamHost we try to be transparent with our customers when it comes to sharing good news and bad news. Whelp, we’ve got some news to share today, and thankfully for both of us it’s all good!

The web is full of nice people and not-so-nice people. We are on constant watch for the not-so-nice people, and there is no shortage of them. We see thousands of attempted hack attempts against our customer database every day (most are automated), and we have been able to intercept and mitigate them all. Hackers want to touch your junk, and it’s our job to keep them at bay.

Hackers have not compromised any personally identifiable account information and our team continues to work constantly and quickly to be proactive in catching these masked bandits. This notice is an effort to share some things we’ve done recently in our efforts to thwart the ever-present hacker threat.

We’ve upped our already upped-security and taken some additional steps to help protect our customers:

• We performed a full audit of all of our legacy systems, removing unnecessarily stored data and improving the encryption of private data where applicable.

• We put new monitoring systems in place to better prepare us to react to any new threats that may arise.

• We’ve rolled out more thorough auditing processes to make certain our systems are kept in a proper configuration from here on out.

• We want to remind you about our IP-Lockdown feature. When it’s turned on in your Web Panel preferences you’ll be asked for approval if anyone attempts to login to your DreamHost control panel from a new IP address.

• We have adjusted our default FTP and SSH user security settings to prevent tampering by other users.   More information on this setting can be found here: http://wiki.dreamhost.com/Enhanced_User_Security

• We are automatically cleaning hacked sites and proactively looking for signs of compromise. As part of this ongoing effort we are sending diagnostic scans to customers and cleaning certain specific hacked injections that we have found in large quantities.

• We are reminding you to update your CMS scripts.

• We’ve added password strength indicators for all passwords entered in the panel to help make sure your password is super strong.

We encourage you to change passwords to your DreamHost control panel and elsewhere on a regular basis. Might as well do it now!

Below are some best practices to make sure you are extra safe:

• Use different passwords for different accounts.

• Pick secure passwords

• Use our password strength indicator

• Change your passwords regularly

Our security team doesn’t sleep so that you can.

Please enjoy your eight hours free of interruption – on us!

-DreamHost Security Team


12:52 pm

Shared web server “izar” not accessible ( RESOLVED )

Posted (March 8th, 2012 at 12:52 pm PST) by Oscar

The shared web server ‘izar’, is currently not accessible. Our system engineering team is working on the issue now, and we will update this post once we have more information. We apologize for the inconvenience this is causing. We will keep this post updated with as much detail as we can provide.

If you have any questions or concerns, please contact Support under Support > Contact Support, in your panel.

Please note, this is regarding the shared web server ‘izar’ and no other services (email, MySQL) or servers are affected. To check if you are on this server, please check your user’s home machine in the DreamHost panel.

Update Mar 8th, 1:30pm PST:
The machine is back online. All services have been restored. If you notice any problems, please contact support and we will gladly take a look!


11:17 am

Connectivity issues to some of our datacenters (RESOLVED)

Posted (March 5th, 2012 at 11:17 am PST) by andrea

We are currently experiencing some connectivity issues to some of our datacenters. Our networking, system engineers and datacenter operations teams have been deployed and are investigating. Sorry about the inconvenience this is causing. Please check back here often, as we will be posting updates as we receive them. – Andrea

Update Mar 5th, 10:30pm PST: As we haven’t seen any more problems related to this issue I’m resolving this post. If you’re having any trouble with your hosting services, it’s most likely unrelated. Please contact support with details, so we can assist you. – Dan J

Previous updates available after the jump →


4:01 am

Web VPS host server ‘homie-vserver172′ moving to new hardware and restoring from backups (RESOLVED)

Posted (March 5th, 2012 at 4:01 am PST) by jj

Due to complications with the drives on the web VPS host server ‘homie-vserver172′, we are moving the server over to new hardware and are restoring all data from our backup server to the new hardware. During this process, the data on the backup server will be mounted directly over NFS to allow all services to remain online for the duration of the restoration.

We apologize for the inconvenience caused by this situation. We will keep this post updated with as much detail as we can provide. If you have any questions or concerns, please contact support with the details.

Please note, this is regarding the web VPS host server ‘homie-vserver172′ and no other services (email, MySQL) or servers are affected. The affected VPS guests on this host can be found in the first comment below this post. To check if you’re on this server, please go to the DreamHost Panel to check your VPS guest number.

Update Mar 10th, 1:07am PST: This restore is now complete. Thank you for your patience during this process! Please contact support if you have any issues with your sites or vps server. -Oscar Previous updates available after the jump →


1:34 am

Shared web server ‘windhoek’ moved to new hardware and restoring from backups (RESOLVED)

Posted (March 4th, 2012 at 1:34 am PST) by jj

The shared web server ‘windhoek’ was submitted to an unscheduled file-system integrity check. This is required due to issues with the RAID on the server switching to read-only mode following a reboot of the server. We expect this process to take a several hours depending on the state of the file-system and we will provide frequent updates below as the check completes. However, we have already brought a new hardware online, and we’re serving your sites from backups, so you should be up and running. We apologize for the inconvenience that this situation causes!

We will keep this post updated with as much detail as we can provide. Please note, this is regarding the shared web server windhoek only, and no other servers or services (such as mail or mysql) have been affected. To check if you’re on this server, please check your user’s home machine in the DreamHost panel. If you have any questions or concerns, please contact support with the details, so we can assist you.

Update Mar 12th, 9:00pm PDT:  We should now have this issue resolved, all but two users are restored. We’ve emailed them directly. If you’re missing anything, please contact support — Brian H

Previous updates available after the jump →


7:11 pm

Shared web server ‘castor’ moved to new hardware and restoring from backups (RESOLVED)

Posted (March 2nd, 2012 at 7:11 pm PST) by Dan J

Due to issues with the RAID controller on the shared web server ‘castor’, we have moved the server on to new hardware and are restoring all data from our backup server to the new hardware. During this process, the data on the backup server will be mounted directly over NFS to allow all services to remain online for the duration of the restoration.

We apologize for the inconvenience caused by this unplanned outage. We will keep this post updated with as much detail as we can provide. If you have any questions or concerns, please contact Support under Support > Contact Support, in your panel.

Please note, this is regarding the shared web server ‘castor’ and no other services (email, MySQL) or servers are affected. To check if you are on this server, please check your user’s home machine in the DreamHost panel.

Update Mar 7th, 11:10am PST: This restore is now complete and all sites are up and resolving. If you notice any issues with your sites or are missing date, please contact support! – Oscar Previous updates available after the jump →


5:55 am

Shared web server ‘port-au-prince’ moved to new hardware and restoring from backups (RESOLVED)

Posted (March 2nd, 2012 at 5:55 am PST) by patrickm

Due to issues with the RAID controller on the shared web server ‘port-au-prince’, we have moved the server on to new hardware and are restoring all data from our backup server to the new hardware. During this process, the data on the backup server will be mounted directly over NFS to allow all services to remain online for the duration of the restoration.

We apologize for the inconvenience caused by this unplanned outage. We will keep this post updated with as much detail as we can provide. If you have any questions or concerns, please contact Support under Support > Contact Support, in your panel.

Please note, this is regarding the shared web server ‘port-au-prince’ and no other services (email, MySQL) or servers are affected. To check if you are on this server, please go to the Users > Manage Users page in your Web Panel and check the “machine” for your user.

Update Mar 7th, 4:54pm PST: The restore is now complete! As mentioned in previous updates, the most recent backup we were able to find on this server was from January 6th. If you have any newer files that have been uploaded since then, they will need to be re-uploaded. Please contact support if you have any questions or problems with sites hosted on this server. Thanks for your patience! – Seohee Previous updates available after the jump →


9:16 pm

Shared web server ‘singapore’ moved to new hardware and restoring from backups (RESOLVED)

Posted (March 1st, 2012 at 9:16 pm PST) by Dan J

Due to issues with the RAID controller on the shared web server ‘singapore’, we have moved the server on to new hardware and are restoring all data from our backup server to the new hardware. During this process, the data on the backup server will be mounted directly over NFS to allow all services to remain online for the duration of the restoration.

We apologize for the inconvenience caused by this unplanned outage. We will keep this post updated with as much detail as we can provide. If you have any questions or concerns, please contact Support under Support > Contact Support, in your Web Panel.

Please note, this is regarding the shared web server ‘singapore’, and no other services (email, MySQL) or web servers are affected. To check if you are on this machine, please go to the Users > Manage Users page in your Web Panel and check the “machine” for your user.

Update Mar 6th, 4:20pm PST: The restore on this server is now complete. We are now resolving this status post. Please contact support if you have any further problems on this machine or have any other questions or issues! -dan w

Previous updates available after the jump →


 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
3:37 pm

Roundcube webmail upgrade tonight, March 1st, at 7pm (PST) (RESOLVED)

Posted (March 1st, 2012 at 3:37 pm PST) by danwy

We will be taking our Roundcube webmail interface offline tonight for a short period of time to upgrade its software. This will only affect your ability to login via the Roundcube webmail client. If you’re using a mail client like Outlook or Thunderbird, or the Squirrelmail webmail interface, you will not be affected. We will update this status post when the upgrade is complete or if there are any unforseen issues that come up. Please keep an eye on this post for updates and contact support with any concerns or problems.

Update Mar 1st, 7:56pm PST: The upgrade is now complete. The RoundCube installation has been upgraded from from version .5.3 to .7.1. This should boost performance as well as allow for some fancy new features. For a full list of changes you can check here. Thanks -Dan J


2:05 pm

Connectivity issues in one of our datacenters

Posted (February 29th, 2012 at 2:05 pm PST) by andrea

Some of our customers may be experiencing connectivity issues right now. This would affect web and mysql services, but only for a small subset of our customers. We apologize for the inconvenience this is causing, and we will be posting updates as soon as we have more information. – Andrea

Update Feb 29th, 2:18pm PST: We are happy to confirm that the situation was quickly identified and has been resolved. There was a slight misconfiguration with our external routing, but it has been corrected. We’re also continuing to investigate to make sure this will not happen again. Sorry about any inconvenience this may have caused. If you’re still experiencing connectivity issues, they are likely unrelated, so please contact support with the details. We’ll be happy to assist. – Andrea


1:03 pm

VPS host machine ‘homie-vserver288′ under DDoS Attack (RESOLVED)

Posted (February 29th, 2012 at 1:03 pm PST) by danwy

Customers on the VPS web server ‘homie-vserver288’ may not be able access their server or sites due to an ongoing denial of service attack. Our admin team is looking into it and working on containing and stopping the attack. Our sincere apologies for any inconvenience this is causing.

Please note, this is regarding the web VPS host server ‘homie-vserver288′ and no other services (email, MySQL, shared) or servers are affected. The affected VPS guests on this host can be found in the first comment below this post. To check if you’re on this server, please go to the DreamHost Panel to check your VPS guest number.

Update Feb 29th, 7:12pm PST: We found the specific VPS that was being attacked. It was shut down and the IP address was null (trash) routed, and the vserver host is now healthy once more. We appreciate your patience while we got this worked out! -Seohee


3:12 pm

Shared web server ‘mystique’ moved to new hardware and restoring from backups (RESOLVED)

Posted (February 28th, 2012 at 3:12 pm PST) by brians

Due to issues with the RAID controller on the shared web server ‘mystique’, we have moved the server on to new hardware and are restoring all data from our backup server to the new hardware. During this process, the data on the backup server will be mounted directly over NFS to allow all services to remain online for the duration of the restoration.

We apologize for the inconvenience caused by this unplanned outage. We will keep this post updated with as much detail as we can provide. If you have any questions or concerns, please contact Support under Support > Contact Support, in your panel.

Please note, this is regarding the shared web server ‘mystique’ and no other services (email, MySQL) or servers are affected. To check if you are on this server, please go to the Users > Manage Users page in your Web Panel and check the “machine” for your user.

We will continue to update this post as information is made available. Please check back for updates.

Additional information: The mystique webserver was upgraded from Debian lenny to Debian squeeze during this hardware replacement process. Please review this status post for additional information if your rails sites are having issues with mysql2 gem or similar. – Justin

Update Mar 2nd, 1:40pm PST: Great news, the restore is done! If you are missing any data at this time, please contact support and we can check the backups for you  and restore any missing data.  -Oscar Previous updates available after the jump →


2:37 pm

Web VPS host server ‘homie-vserver19′ moving to new hardware and restoring from backups (RESOLVED)

Posted (February 27th, 2012 at 2:37 pm PST) by Brandon

Due to complications with the drives on the web VPS host server ‘homie-vserver19′, we are moving the server over to new hardware and are restoring all data from our backup server to the new hardware. During this process, the data on the backup server will be mounted directly over NFS to allow all services to remain online for the duration of the restoration.

We apologize for the inconvenience caused by this situation. We will keep this post updated with as much detail as we can provide. If you have any questions or concerns, please contact support with the details.

Please note, this is regarding the web VPS host server ‘homie-vserver19′ and no other services (email, MySQL) or servers are affected. The affected VPS guests on this host can be found in the first comment below this post. To check if you’re on this server, please go to the DreamHost Panel to check your VPS guest number. -Brandon

Update Mar 5th, 5:15pm PST:We’re happy to report that this incident is now resolved. All guests are restored, but if you’re missing any data or are having issues with your sites, please don’t hesitate to contact support with the details. – Andrea

Previous updates available after the jump →


10:28 am

Web VPS host homie-vserver142 moving to new hardware and restoring from backups (RESOLVED)

Posted (February 27th, 2012 at 10:28 am PST) by brians

Due to issues with the RAID controller on the web VPS host homie-vserver142, we have moved the server on to new hardware and are restoring all data from our backup server to the new hardware. During this process, the data on the backup server will be mounted directly over NFS to allow all services to remain online for the duration of the restoration.

We apologize for the inconvenience caused by this unplanned outage. We will keep this post updated with as much detail as we can provide. If you have any questions or concerns, please contact Support under Support > Contact Support in your panel.

Please note, this is regarding the VPS host machine ‘homie-vserver142’ and no other services (email, MySQL) or machines are affected. The affected VPS guests on this host can be found in the first comment below this post. To check if you’re on this server, please go to the DreamHost Panel to check your VPS guest number.

We will continue to update this post as information is made available. Please check back for updates.

Update Mar 14th, 2:50pm PDT: All VPS guests have been restored now! Your sites should be working, but if you have any concerns or questions, please contact support with the details. – Oscar Previous updates available after the jump →


8:35 pm

Web VPS host server ‘homie-vserver234′ moving to new hardware and restoring from backups (RESOLVED)

Posted (February 25th, 2012 at 8:35 pm PST) by Brian@dreamhost

Due to issues with the drives on the web VPS host server ‘homie-vserver234′, we have moved the server on to new hardware and are restoring all data from our backup server to the new hardware. During this process, the data on the backup server will be mounted directly over NFS to allow all services to remain online for the duration of the restoration.

We apologize for the inconvenience caused by this unplanned outage. We will keep this post updated with as much detail as we can provide. If you have any questions or concerns, please contact Support under Support > Contact Support, in your panel.

Please note, this is regarding the web VPS host server ‘homie-vserver234′ and no other services (email, MySQL) or servers are affected. To check if you are on a VPS guest from this host, please go to the Users > Manage Users page in your Web Panel and check the “machine” for your user. The affected VPS guests can be found in the first comment below this post. — Brian

Update Mar 3rd, 8:50pm PST:  Ok, we’ve received word that this server should now be fully restored. We’re marking this issue as resolved, so if you’re still seeing missing files, please let support know ASAP, and we can look into the issue for you! — Brian H

Previous updates available after the jump →


7:40 pm

Web VPS host server ‘homie-vserver173′ moving to new hardware and restoring from backups (RESOLVED)

Posted (February 25th, 2012 at 7:40 pm PST) by Brian@dreamhost

Due to issues with the drives on the web VPS host server ‘homie-vserver173′, we have moved the server on to new hardware and are restoring all data from our backup server to the new hardware. During this process, the data on the backup server will be mounted directly over NFS to allow all services to remain online for the duration of the restoration.

We apologize for the inconvenience caused by this unplanned outage. We will keep this post updated with as much detail as we can provide. If you have any questions or concerns, please contact Support under Support > Contact Support, in your panel.

Please note, this is regarding the web VPS host server ‘homie-vserver173′ and no other services (email, MySQL) or servers are affected. To check if you are on a VPS guest from this host, please go to the Users > Manage Users page in your Web Panel and check the “machine” for your user. The affected VPS guests can be found in the first comment below this post. — Brian

Update Mar 17th, 4:20pm PDT: The restore is now complete for all guest machines. If you have any problems with your VPS guest or are missing data,  please let support know. – Oscar Previous updates available after the jump →


11:41 pm

Web VPS host ‘homie-vserver127′ moved to new hardware and restoring from backups ( RESOLVED )

Posted (February 23rd, 2012 at 11:41 pm PST) by chalise

Due to issues with the RAID controller on the web VPS host ‘homie-vserver127′, we have moved the server on to new hardware and are restoring all data from our backup server to the new hardware. During this process, the data on the backup server will be mounted directly over NFS to allow all services to remain online for the duration of the restoration.

We apologize for the inconvenience caused by this unplanned outage. We will keep this post updated with as much detail as we can provide. If you have any questions or concerns, please contact Support under Support > Contact Support, in your panel.

Please note, this is regarding the web VPS host ‘homie-vserver127′ and no other services (email, MySQL) or servers are affected. To check if you are on a VPS guest from this host, please go to the Users > Manage Users page in your Web Panel and check the “machine” for your user. The affected VPS guests can be found in the first comment below this post.

Update Feb 28th, 1:23pm PST: We have confirmation that the restore to the new hardware has finished. All sites should be up. As we used the last available backups for he restore, it’s possible that you may not see the same data that was on the old hardware. If that’s the case and you need newer data but don’t have your own backups, please contact support with the details, so we can assist you. -Brian S

Previous updates available after the jump →


12:28 pm

Shared web server ‘stampeders’ restoring from backups (RESOLVED)

Posted (February 23rd, 2012 at 12:28 pm PST) by brians

Due to corrupt operating system data on the shared web server ‘stampeders’, we have moved the server on to new hardware. We are restoring all data from our backup server to this new machine. At this time, only operating system data is affected, all customer data shouldn’t be affected at all. During this process, the data on the backup server will be mounted directly over NFS to allow all services to remain online for the duration of the restoration.

We apologize for the inconvenience caused by this unplanned outage. We will keep this post updated with as much detail as we can provide. If you notice any problems, please let us know via our Support system under Contact Support, in your panel.

Please note, this is regarding the shared web server ‘stampeders’ and no other services (email, MySQL) or servers are affected. To check if you are on this server, please go to the Users > Manage Users page in your Web Panel and check the “machine” for your user.

Update Mar 2nd, 1:50pm PST: Great news! The restore has now finished on this server! If you are missing any data at this time, please contact support and we can check the backups for you  and restore any missing data. – Oscar

Previous updates available after the jump →


 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
8:50 am

Web VPS host ‘homie-vserver131′ back up and running on original hardware ( RESOLVED )

Posted (February 23rd, 2012 at 8:50 am PST) by Oscar

Due to issues with the RAID controller on the Web VPS host ‘homie-vserver131′, we are moving the server on to new hardware and are restoring all data from our backup server to the new server. During this process, the data on the backup server will be mounted directly over NFS to allow all services to remain online for the duration of the restoration.

We apologize for the inconvenience caused by this unplanned outage. We will keep this post updated with as much detail as we can provide. If you have any questions or concerns, please contact Support under Support > Contact Support, in your panel.

Please note, this is regarding the Web VPS host machine ‘homie-vserver131’ and no other services (email, MySQL) or servers are affected. To check if you are on a VPS guest from this host, please go to the Users > Manage Users page in your Web Panel and check the “machine” for your user. The affected VPS guests can be found in the first comment below this post.

Update Feb 24th, 8:40am PST: Our Admin team has gotten the original machine back in working order. As the original hardware is now running without issue, guests have been started back up and don’t need to be moved to new hardware. All affected VPS guests should now be online, but please Contact Support if you’re still experiencing issues. -Brandon

Previous updates available after the jump →


11:32 pm

Shared web server ‘michaelkelly’ being moved to new hardware and restoring from backups (RESOLVED)

Posted (February 22nd, 2012 at 11:32 pm PST) by chalise

Due to issues with the RAID controller on the shared web server ‘michaelkelly’, we have moved the server on to new hardware and are restoring all data from our backup server to the new hardware. During this process, the data on the backup server will be mounted directly over NFS to allow all services to remain online for the duration of the restoration.

We apologize for the inconvenience caused by this unplanned outage. We will keep this post updated with as much detail as we can provide. If you have any questions or concerns, please contact Support under Support > Contact Support, in your panel.

Please note, this is regarding the shared server ‘michaelkelly’ and no other services (email, MySQL) or servers are affected. To check if you are on this server, please go to the Users > Manage Users page in your Web Panel and check the “machine” for your user.

Update Feb 28th, 5:50pm PST: The restore is now complete. All data from the backups and from the old hardware has been copied over.  If you are missing any data, please contact support detailing what you are missing and we can restore it for you. Please be sure to Contact Support if you have any questions or concerns. – Oscar Previous updates available after the jump →


5:09 pm

Web VPS host server ‘homie-vserver89′ restoring from backups to new hardware (RESOLVED)

Posted (February 22nd, 2012 at 5:09 pm PST) by Dan J

Due to issues with the RAID controller on the web VPS host server ‘homie-vserver89′, we have moved the server on to new hardware and are restoring all data from our backup server to the new hardware. During this process, the data on the backup server will be mounted directly over NFS to allow all services to remain online for the duration of the restoration.

We apologize for the inconvenience caused by this unplanned outage. We will keep this post updated with as much detail as we can provide. If you have any questions or concerns, please contact Support under Support > Contact Support, in your panel.

Please note, this is regarding the web VPS host server ‘homie-vserver89′ and no other services (email, MySQL) or servers are affected. To check if you are on a VPS guest from this host, please go to the Users > Manage Users page in your Web Panel and check the “machine” for your user. The affected VPS guests can be found in the first comment below this post.

Update Feb 22nd, 8:52pm PST: Our admin team have decided that the server appears to be healthy now and this was just a false alarm. We do not need to move homie-vserver89 to new hardware at this time. -Dan J

Update Feb 22nd, 6:08pm PST: Our admin team were able to get the old hardware online again and are monitoring the server it to determine if they need to move the server to new hardware. All VPS are currently online. We’ll have more info in a little while. -Dan J

 


3:39 pm

Scheduled maintenance for Ajaxplorer (WebFTP) tonight @ 8pm PST (RESOLVED)

Posted (February 22nd, 2012 at 3:39 pm PST) by Seohee

We will be upgrading Ajaxplorer (WebFTP) tonight, which is estimated to take about an hour. This is regarding the WebFTP link found at ‘Manage Domains’ in the DreamHost control panel. During this time, it is possible that WebFTP will not be functional. You may want to use a different (non-web-based) FTP client during that time. This wiki article will give you further details on other available FTP client software.

We appreciate your patience and are sorry for the inconvenience. We’ll make sure to update this post once the upgrade begins.

Update Feb 22nd, 8:29pm PST: The Ajaxplorer update has completed. Everything appears to be working. If you find any problems then please contact support. -Dan J


12:58 pm

Shared web server ‘cayenne’ restoring from backups to new hardware (RESOLVED)

Posted (February 21st, 2012 at 12:58 pm PST) by Oscar

Due to issues with the RAID controller on the shared web server ‘cayenne’, we have moved the server on to new hardware and are restoring all data from our backup server to the new hardware. During this process, the data on the backup server will be mounted directly over NFS to allow all services to remain online for the duration of the restoration.

We apologize for the inconvenience caused by this unplanned outage. We will keep this post updated with as much detail as we can provide. If you have any questions or concerns, please contact Support under Support > Contact Support, in your panel.

Please note, this is regarding the shared web server ‘cayenne’ and no other services (email, MySQL) or servers are affected. To check if you are on this server, please go to the Users > Manage Users page in your Web Panel and check the “machine” for your user.

Update Feb 23rd, 2:05am PST: The restore on this server has completed and all sites are now loading from the new hardware! If you notice missing or outdated data, or experience any other issues, please be sure to contact support. – Chalise

Previous updates available after the jump →


3:52 am

Web VPS host server ‘homie-vserver207′ restoring from backups to new hardware ( RESOLVED )

Posted (February 18th, 2012 at 3:52 am PST) by patrickm

Due to issues with the RAID controller on the web VPS host server ‘homie-vserver207′, we have moved the server on to new hardware and are restoring all data from our backup server to the new hardware. During this process, the data on the backup server will be mounted directly over NFS to allow all services to remain online for the duration of the restoration.

We apologize for the inconvenience caused by this unplanned outage. We will keep this post updated with as much detail as we can provide. If you have any questions or concerns, please contact Support under Support > Contact Support, in your panel.

Please note, this is regarding the web VPS host server ‘homie-vserver207′ and no other services (email, MySQL) or servers are affected. To check if you are on a VPS guest from this host, please go to the Users > Manage Users page in your Web Panel and check the “machine” for your user. The affected VPS guests can be found in the first comment below this post.

The affected VPS guests can be found in the first comment below this post.

Update Feb 22nd, 2:56pm PST: We have received final word from the admins, that the restore from backups has completed. If you are seeing any issues with your data, please contact support so we can assist in restoring it from the old hardware. -Brian S

Previous updates available after the jump →


4:31 pm

Shared web server ‘paramaribo’ offline for RAID card replacement (RESOLVED)

Posted (February 17th, 2012 at 4:31 pm PST) by Dan J

We apologize for the inconvenience, but we had to take the shared web server ‘paramaribo’ offline briefly for a RAID card replacement.  We expect this maintenance to take approximately 20 minutes.  We will update this post when the issue has been resolved!

Please note, this is regarding the shared web server ‘paramaribo’ and no other services (email, MySQL) or machines are affected. To check if you’re on this server, please go to the DreamHost panel and check the home server for your user on the “Users > Manage Users” page.

Update Feb 17th, 4:50pm PST: The RAID card has been replaced and this server is now back online. If you continue to have issues then please contact support. -Dan J


4:03 pm

Web VPS host homie-vserver173 offline for file-system check (RESOLVED)

Posted (February 17th, 2012 at 4:03 pm PST) by Dan J

The web VPS server homie-vserver173 is offline for an unscheduled file-system integrity check. This is required due to issues with the RAID on the server switching to read-only mode following a reboot of the server. We expect this process to take about two hours and we will provide frequent updates below as the check completes. We apologize for the inconvenience that this situation causes!

We will keep this post updated with as much detail as we can provide. If you have any questions or concerns, please contact Support under Support > Contact Support, in your panel.

A list of the affected VPS guests can be found in the first comment below this post.

Update Feb 17th, 4:50pm PST: The file-system check has completed and this server is now back online. All VPS on this machine should be online as well. If you continue to have issues then please contact support. -Dan J


3:09 am

Web VPS host homie-vserver197 restoring from backups to new hardware ( RESOLVED )

Posted (February 17th, 2012 at 3:09 am PST) by chalise

Due to issues with the RAID controller on the web VPS server ‘homie-vserver197′, we have moved the server on to new hardware and are restoring all data from our backup server to the new hardware. During this process, the data on the backup server will be mounted directly over NFS to allow all services to remain online for the duration of the restoration.

We will keep this post updated with as much detail as we can provide. If you have any questions or concerns, please contact Support under Support > Contact Support, in your panel.

Please note, this is regarding the web VPS server ‘homie-vserver197′ and no other services (email, MySQL) or servers are affected. To check if you are on a VPS guest from this host, please go to the Users > Manage Users page in your Web Panel and check the “machine” for your user. The affected VPS guests can be found in the first comment below this post.

A list of the affected VPS guests can be found in the first comment below this post.

Update Feb 22nd, 2:58pm PST: We have received final word from the admins, that the restore from backups has completed. If you are seeing any issues with your data, please contact support so we can assist in restoring it from the old hardware. -Brian S

Previous updates available after the jump →


12:54 pm

Mailservers on Trend Micro RBL, working on removal from list (RESOLVED)

Posted (February 14th, 2012 at 12:54 pm PST) by brians

We’re well aware of the email issues that you may be experiencing. Unfortunately Trend Micro, maintainer of an RBL (real-time blacklist,) has blacklisted several of our mail servers. Customers of service providers who subscribe to this RBL will not be able to receive email from DreamHost customers until this is resolved.

After several days (nearly a week) of relentlessly trying to get a response from Trend Micro we have finally heard back from them and are processing through a list of their concerns as quickly as possible.

We are hopeful that when we resolve these outstanding issues Trend Micro will take action in a timely manner – and mail service to the affected service providers should then resume.

We apologize for the inconvenience this has caused and appreciate your continued patience. -Brian S

Update March 1st, 5:33pm PST: The mailman server has been unblocked as well, so we’re resolving this case. If you run into any mail blocks, please submit a ticket through the DreamHost panel including the full bounce message with the date and time, so we can address it. – Andrea

Update Feb 28th, 5:33pm PST: We are continuing to get this case fully resolved. We are at the brink of getting that one discussion list unblocked. We appreciate your continued patience. – Andrea

Update Feb 27th, 12:42pm PST: Unfortunately, there is no change in the status below, just letting you know we’re still working on it. We appreciate your continued patience. – Andrea

Update Feb 26th, 2:15pm PST: One of our discussion list servers and our autoresponder server is still blocked. We’re continuing to cooperate with Trend Micro to get this resolved, and working on further reviewing our outgoing mail system to avoid potential issues in the future. We appreciate your patience. If you’re continuing to see issues unrelated to a discussion list or an autoresponder, please submit a ticket through the DreamHost panel including the full bounce message with the date and time. Your case may be unrelated (other block providers), but we would like to take a closer look. – Andrea

Previous updates available after the jump →


 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
6:41 am

Web VPS host homie-vserver158 being moved to new hardware and restoring from backups ( RESOLVED )

Posted (February 13th, 2012 at 6:41 am PST) by Justin@DH

Due to issues with the RAID controller on the Web VPS host server ‘homie-vserver158′, we are moving the server on to new hardware and are restoring all data from our backup server to the new hardware. During this process, the data on the backup server will be mounted directly over NFS to allow all services to remain online for the duration of the restoration. We will update this post as soon as the new hardware is up and the backup server has been mounted.

We apologize for the inconvenience caused by this unplanned outage. We will keep this post updated with as much detail as we can provide. If you have any questions or concerns, please contact Support under Support > Contact Support, in your panel.

Please note, this is regarding the Web VPS host machine ‘homie-vserver158’ and no other services (email, MySQL) or servers are affected. To check if you are on a VPS guest from this host, please go to the Users > Manage Users page in your Web Panel and check the “machine” for your user. The affected VPS guests can be found in the first comment below this post.

Update Feb 22nd, 2:59pm PST: We have received final word from the admins, that the restore from backups has completed. If you are seeing any issues with your data, please contact support so we can assist in restoring it from the old hardware. -Brian S

Previous updates available after the jump →


6:24 am

Web VPS host homie-vserver206 being moved to new hardware and restoring from backups (RESOLVED)

Posted (February 9th, 2012 at 6:24 am PST) by Justin@DH

Due to issues with the RAID controller on the Web VPS host server ‘homie-vserver206′, we are moving the server on to new hardware and are restoring all data from our backup server to the new hardware. During this process, the data on the backup server will be mounted directly over NFS to allow all services to remain online for the duration of the restoration. We will update this post as soon as the new hardware is up and the backup server has been mounted.

We apologize for the inconvenience caused by this unplanned outage. We will keep this post updated with as much detail as we can provide. If you have any questions or concerns, please contact Support under Support > Contact Support, in your panel.

Please note, this is regarding the Web VPS host machine ‘homie-vserver206’ and no other services (email, MySQL) or servers are affected. To check if you are on a VPS guest from this host, please go to the Users > Manage Users page in your Web Panel and check the “machine” for your user. The affected VPS guests can be found in the first comment below this post.

Update Feb 14th, 12:00pm PST: We have received final word from the admins, that the restore from backups has completed. If you are seeing any issues with your data, please contact support so we can assist in restoring it from the old hardware. -Justin

Previous updates available after the jump →


3:06 pm

Mail bouncing for subset of customers (RESOLVED)

Posted (February 7th, 2012 at 3:06 pm PST) by Brian@dreamhost

At this time, we’re seeing some mail bounce for a subset of our customers. We’re in the process of identifying which customers are affected, and what the exact nature of the problem is. As we get info, we’ll update this post. We apologize for the problems this causes! — Brian H

Update Feb. 7th 04:30pm PST: This issue should now be resolved. All mail services are back on and working normally, and any queued messages should now be delivered. If you’re still seeing problems, please contact Support under Support > Contact Support, in your panel. . — Brian H

Previous updates available after the jump →


12:08 pm

Connectivity issues with our Irvine Datacenter (RESOLVED)

Posted (February 4th, 2012 at 12:08 pm PST) by Oscar

We are currently experiencing connectivity issues with our Irvine datacenter. The other datacenters are not being affected.  Our network engineers are currently looking into the issue and we will continue to update this post as soon as we have more information.

Update Feb. 4th 12:24pm: Packet loss from one of our providers was the issue we were seeing. This is resolved now through the joint efforts of our Network team and and our service provider. -Ralph


5:07 pm

Web VPS host homie-vserver163 being moved to new hardware and restoring from backups (RESOLVED)

Posted (February 3rd, 2012 at 5:07 pm PST) by Dan J

Due to issues with the RAID controller on the Web VPS host server ‘homie-vserver163′, we are moving the server on to new hardware and are restoring all data from our backup server to the new hardware. During this process, the data on the backup server will be mounted directly over NFS to allow all services to remain online for the duration of the restoration. We will update this post as soon as the new hardware is up and the backup server has been mounted.

We apologize for the inconvenience caused by this unplanned outage. We will keep this post updated with as much detail as we can provide. If you have any questions or concerns, please contact Support under Support > Contact Support, in your panel.

Please note, this is regarding the Web VPS host machine ‘homie-vserver163’ and no other services (email, MySQL) or servers are affected. To check if you are on a VPS guest from this host, please go to the Users > Manage Users page in your Web Panel and check the “machine” for your user. The affected VPS guests can be found in the first comment below this post.

Update Feb 6th, 11:30am PST: Thanks for your patience. The move to the new hardware is now finished. All sites should be operational, so if you have any questions or concerns, please contact support with the details. – Andrea

Previous updates available after the jump →


8:54 am

Web VPS host homie-vserver168 being moved to new hardware and restoring from backups (RESOLVED)A

Posted (February 3rd, 2012 at 8:54 am PST) by Oscar

Due to issues with the RAID controller on the Web VPS host server ‘homie-vserver168′, we are moving the server on to new hardware and are restoring all data from our backup server to the new hardware. During this process, the data on the backup server will be mounted directly over NFS to allow all services to remain online for the duration of the restoration. We will update this post as soon as the new hardware is up and the backup server has been mounted.

We apologize for the inconvenience caused by this unplanned outage. We will keep this post updated with as much detail as we can provide. If you have any questions or concerns, please contact Support under Support > Contact Support, in your panel.

Please note, this is regarding the Web VPS host machine ‘homie-vserver168’ and no other services (email, MySQL) or servers are affected. To check if you are on a VPS guest from this host, please go to the Users > Manage Users page in your Web Panel and check the “machine” for your user. The affected VPS guests can be found in the first comment below this post.

Update Feb 6th, 11:30am PST: Thanks for your patience. The move to the new hardware is now finished. All sites should be operational, so if you have any questions or concerns, please contact support with the details. – Andrea

Previous updates available after the jump →


10:38 am

Shared web server ‘portvila’ not accessible (RESOLVED)

Posted (February 2nd, 2012 at 10:38 am PST) by Oscar

The shared web server “portvila” is currently not accessible. It did not reboot correctly after having its raid card replaced. Our system administration team is working on the issue and we will update this post as soon as we have further information.

Please note, this is regarding the shared web server ‘portvila’ and no other services (email, MySQL) or servers are affected. To check if you are on this server, please go to the Users > Manage Users page in your Web Panel and check the “machine” for your user. – Oscar

Update Feb 2nd, 10:40am PST: The raid array has finished rebuilding and all sites and services on ‘portvila’ are back up and running normally. Thank you for your patience! If you have any further issues, please be sure to contact support Support > Contact Support, in your panel. – Oscar


8:37 am

Email connectivity issue in one of our datacenters (RESOLVED)

Posted (February 2nd, 2012 at 8:37 am PST) by Oscar

We have received reports of connectivity issues to our mail service in one of our datacenters. It manifested in slowness when sending or receiving and mail timeouts from both webmail and email clients. Our system adminstration team is investigating the cause and is working diligently towards a resolution.  We are already seeing improvements, but we will continue to monitor the situation as long as necessary. We’ll update this post when we receive further information. Your patience and understanding is appreciated, and we apologize for the inconvenience.

Please note, this did not affect mail sending from web servers, or any other service such as web and mysql. – Oscar

Update Feb 2nd, 9:13am PST: The cause of the above problems appears to be a slight connectivity ‘hiccup’ between the file storage machine and the email processing servers. We double checked for hardware issues, and it checks out fine. We will continue to monitor the connection and investigate more about that failure.
We’re resolving this case, but if you need further assistance, please don’t hesitate to contact support with the details, so we can assist you. If you already emailed about this, you will receive a reply shortly. – Andrea


7:45 pm

Scheduled emergency network maintenance tonight at 8pm PST (RESOLVED)

Posted (January 30th, 2012 at 7:45 pm PST) by Seohee

Network maintenance is scheduled to start at 8pm PST tonight to update some routes as they are causing additional problems during our network outages. This should cause no downtime at all, but please do contact support if you run into any issues.

Update Jan 30th, 10:04pm PST: The maintenance has completed now. The admins are currently verifying connectivity and route advertisement. We haven’t seen any reports of downtime but if you are having trouble then please contact support. -Dan J

Previous updates available after the jump →


1:16 pm

Shared web server Sinistra having issues (RESOLVED)

Posted (January 30th, 2012 at 1:16 pm PST) by danwy

The shared web server ‘sinistra’ is having some issues at the moment. It seems like there may be some php packages missing on it due to the recent issues we’ve been having, but we’re not certain yet. This is causing sites hosted on it to give 500 errors and give ‘premature end of script headers’ errors in the domains error logs. Our admins are checking on this server and are working on getting it back up and running smoothly. We’ll update this status post shortly once we have some more information regarding its stability restoration process.

Please note, this post is only regarding the sinistra web server. No other web servers or other services, such as email or mysql have been affected. To find out if you’re on this server, please to to the panel and check the machine listed for your FTP/Shell user. – DanW

Update Feb 1st, 4:02pm PST: We’re happy to report that sinistra has been stable for a while now, and its load is in the normal range. We’re confident that the offloads have helped to resolve this problem, but we will continue to montior the server for a little while longer.
We apologize for all inconvenience the instability has caused. – Andrea

Update Feb 1st, 11:34am PST: Our system administrators continue moving users off this server to alleviate the load that has been slowing things down lately on your websites. They are also closely monitoring to find balance between how fast they can offload to lower the load and not cause too much strain at the same time. Unfortunately, the machine had to be rebooted again earlier this morning, but it has been stable ever since.
We appreciate your continued patience. Please check back for updates. -Andrea

Previous updates available after the jump →


 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
8:09 am

Connectivity issues in one of our datacenters ( RESOLVED )

Posted (January 30th, 2012 at 8:09 am PST) by Justin@DH

As of 8am PST, we have been experienced connectivity issues in our Downtown Los Angeles datacenter. The other datacenters aren’t affected. Our system administration and datacenter operation team was engaged right away and promptly restored some of the services. Our network engineers are also looking into the problem, and we are continuing to fix up remaining issues and investigate possible causes.

Update Jan 30th, 10:15am PST: Our sincere apologies for the intermittent network outage this morning especially coming on the heels of the downtime experienced yesterday. All services have been restored.

We’ve investigated the cause of the network outage and determined that a distributed denial of service (DDOS) attack severely impaired a network link between our border and core routers. We have DDOS mitigation systems in place, but given the increase in activity in 2012, we’d already been testing new more powerful systems to implement within the next month. We’ll be accelerating this implementation.

We will be scheduling an 8pm PST emergency network maintenance window tonight to make some network configuration changes that will mitigate against DDOS attacks in the short term.

Please contact support for any further assistance needed. We will be creating a separate status post for the maintenance window tonight. – Ralph

Previous updates available after the jump →


1:32 am

Easy/Simple One Click Installs temporarily offline ( RESOLVED )

Posted (January 30th, 2012 at 1:32 am PST) by chalise

Our admin team has had to temporarily take our Easy/Simple One Click Install services offline.

To see if you are using an Easy/Simple One Click Install, go to Goodies / One Click Installs in your panel and click “Manage Installed Applications”. Any installation that says “Simple One Click App” to the right of it will be affected. This will not affect any other One Click Install services we offer.

We will update this post again when our admin team is able to bring this service back online. Thanks for your patience!

Update Jan 30th, 4:03pm PST:Our admins have informed us that the Easy/Simple One Click Installs are back online. Sorry for any inconvenience this has caused. Your patience is greatly appreciated. If you continue to encounter problems with your one-click software, please contact Tech Support with the details. -Brian S


12:14 pm

Webmail login issues (RESOLVED)

Posted (January 29th, 2012 at 12:14 pm PST) by danwy

We’re currently experiencing some problems with one of our databases that services our webmail machines. As a result, you may be experiencing intermittent login issues when trying to login via webmail. We initially thought that this issue may have been related to another posted outage, but upon discovering this was its own issue we decided to make its own status post. Our admins are investigating the cause and will work on resolving this ASAP. We’ll update this post as soon as we have any further information regarding it.

Update Jan 29th, 1:40pm PST: These issues are resolved now. We needed to restart dovecot on our webmail machines and have not been seeing any login errors since that was done. We’re not quite sure what caused this to prevent the database from connecting, but we’ll be investigating that further. We’ll be resolving this post now, as we have not been able to duplicate any errors since restarting the dovecot service. If you continue to have problems with ‘unknown username or password’ errors, please contact support to have them investigate it further for you. -dan w


3:35 am

Web, SSH, and FTP services for a subset of VPS, shared, and dedicated machines down (RESOLVED)

Posted (January 29th, 2012 at 3:35 am PST) by jj

The apache (web), SSH, and ftp, and services on a subset of our VPS and dedicated servers are currently down. FTP services on some shared servers are also experiencing downtime. Our system administration and data center operation teams are currently on the case and we are attempting to restore services as soon as possible. Further updates will be provided.

Please note that this only affects web VPS/dedicated and shared web server FTP services, no other services or servers (MySQL or mail, for example) are affected.

Update Feb 1st, 4:30pm PST: We are no longer seeing any fall outs from Sundays’downtime, so we consider this incident resolved. Thank you again for your patience and understanding while we were working on this. If you have any questions about this case or need further assistance, please contact support, so we can assist you. – Andrea

Previous updates available after the jump →


6:31 pm

Maintenance scheduled for MySQL server ‘madal’ on Feb 5th @ 9pm PST (RESOLVED)

Posted (January 26th, 2012 at 6:31 pm PST) by Dan J

We’ve scheduled a window for maintenance on the MySQL server ‘madal’. The maintenance will begin Sunday, Feb 5th at 9pm PST. The window should be limited to 30 minutes or so during this time and will only affect MySQL related services. It will only affect site availability if the site is using a database that is hosted on this server. We will update this post shortly before the maintenance window starts and update it when it’s complete. If you have any questions regarding this, please don’t hesitate to contact support to let us know.

Update Feb 5th, 9:24pm PST: The maintenance has completed successfully and the server was only down for about 15 minutes. We appreciate your patience and hope this improves the server’s performance. If you run into any problems, please contact support and let us know! -Seohee


2:40 pm

Connectivity issues to a small subset of web servers (RESOLVED)

Posted (January 26th, 2012 at 2:40 pm PST) by andrea

Currently, we’re experiencing connectivity issues to a small subset of web servers in one of our datacenters. Our Datacenter Operations team is investigating the problem, and we will post details when we know more.

The affected servers are:
pigwidgeon eskimos homie-vserver90 alouettes bluebombers tigercats argonauts stampeders roughriders olympiacos herediano d-fenders bluejays homie-vserver89 homie-vserver91 homie-vserver92 homie-vserver93 homie-vserver94 bandar fever revolution saprissa fire crew rapids earthquakes shock liberty monarchs islanders blackhawks predators lightning thrashers flames dynamo

You can tell if you’re affected by checking ‘Your Web Server’ under Account Status on the upper right corner of your DreamHost panel. If you’re on a VPS and are experiencing downtime, please check the upcoming update for the list of VPS guests affected.
Please note, this does not involve any other machines, or any other services such as mail or mysql. – Andrea

Update Jan 25th, 3:05pm PST: We just received confirmation that the problem with the above servers is now resolved. We had a temporary issue with the switch servicing those machines: it wasn’t able to reach the internet. It is now fixed. We apologize for the inconvenience this has caused. -Andrea

Previous updates available after the jump →


8:39 pm

Shared web server ‘torr’ moved to new hardware and restoring from backups (RESOLVED)

Posted (January 23rd, 2012 at 8:39 pm PST) by Dan J

Due to issues with the RAID controller on the shared web server ‘torr’, we have moved the server on to new hardware and are restoring all data from our backup server to the new hardware. During this process, the data on the backup server will be mounted directly over NFS to allow all services to remain online for the duration of the restoration.

We apologize for the inconvenience caused by this unplanned outage. We will keep this post updated with as much detail as we can provide. If you have any questions or concerns, please contact Support under Support > Contact Support, in your panel.

Please note, this is regarding the shared web server torr, and no other services (email, MySQL) or web servers are affected. To check if you are on this machine, please go to the Users > Manage Users page in your Web Panel and check the “machine” for your user.

Update Jan 25th, 9:58pm PST: The restore process has finished! If your site is missing any recent content then please contact support and we can attempt to restore the data from the old hardware. Thanks! -Dan J

Previous updates available after the jump →


4:41 pm

Degraded performance on personal backup server hanjin (RESOLVED)

Posted (January 22nd, 2012 at 4:41 pm PST) by danwy

The hanjin personal backup server has been experiencing some recent stability issues, which includes access difficulties and timeouts. Some solutions have been discussed by our admin team, but we haven’t decided on the best plan of attack yet. We wanted to put up this post to let our customers know that we are aware of the problems with this service, but unfortunately we don’t have an ETA on when they will be resolved at this time.

To clarify this post is regarding the personal backup server we offer as an additional service for all our customers. More information about what this service is for is on our DreamHost panel. This post is regarding only about the difficulties of accessing or staying connected to that personal backup service, and is not related to any other services or machines. This is not related to backups of your web, mysql or email services.

Update Jan 25th, 2:50pm PST: The two changes we made yesterday proved to be successful. We are now confident to close this case; all feedback from customers have been positive indicating the personal backup service is running again as expected. Thank you for your patience while we were working on this.
If you’re still having issues accessing or using the service while logged in, it’s most likely unrelated. Please contact support with the details, so we can look into it for you. -Andrea

Previous updates available after the jump →


3:44 pm

Shared web server ‘albali’ under file system check and being moved to new hardware (RESOLVED)

Posted (January 22nd, 2012 at 3:44 pm PST) by andrea

Those of you on shared web server albali have been experiencing on-and-off downtime today. We’ve been trying to fix the machine, but decided that a file system check was necessary when the machine came back up as read-only (basically, mostly unusable for our customers). While that file system check is running, we’re also currently initiating a move to new hardware based on last backups, so we can serve sites in the meantime.

Please note, this is regarding the shared web server albali, and no other services (email, MySQL) or web servers are affected. To check if you are on this machine, please go to the Users > Manage Users page in your Web Panel and check the “machine” for your user.

We apologize for the inconvenience this is causing. Please check back here often for updates. – Andrea

Update Jan 26th, 1:00pm PST: The restore is now complete and all sites should be up and running. Should you have any issues with your sites being down or missing data, please contact our support team and we will be glad to look into it for you.  – Oscar Previous updates available after the jump →


1:07 am

Web VPS host homie-vserver233 moved to new hardware and restoring from backups (RESOLVED)

Posted (January 22nd, 2012 at 1:07 am PST) by jj

Due to issues with the RAID controller on the Web VPS host server ‘homie-vserver233′, we have moved the server on to new hardware and are restoring all data from our backup server to the new hardware. During this process, the data on the backup server will be mounted directly over NFS to allow all services to remain online for the duration of the restoration.

We apologize for the inconvenience caused by this unplanned outage. We will keep this post updated with as much detail as we can provide. If you have any questions or concerns, please contact Support under Support > Contact Support, in your panel.

Please note, this is regarding the Web VPS host machine ‘homie-vserver233’ and no other services (email, MySQL) or servers are affected. To check if you are on a VPS guest from this host, please go to the Users > Manage Users page in your Web Panel and check the “machine” for your user. The affected VPS guests can be found in the first comment below this post.

Update Jan 29th, 5:15pm PST: We have confirmed with our admin team that the restore for this server has completed. If your sites are missing data that means the restore did not find anything in the backups automatically.  If you find you are missing data, or have old data on your VPS after the restore is done, please contact support and we will be happy to help see if there is anything else we can do for you. -dan w

Previous updates available after the jump →


 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
9:22 am

Password changes delayed in the web panel (RESOLVED)

Posted (January 21st, 2012 at 9:22 am PST) by Oscar

Processing user updates is taking long than usual due to the of sheer number of customers requesting password changes on our system. We understand your desire to get things working in an expeditious manner and we are working hard to get you there. We’re examining ways of decreasing the queue depth but we’re still faced with the fact that there is a considerable amount of work to be processed and apologize for the delay.

Update Jan 22nd, 9:12pm PST: We’ve been keeping a close eye on our panel update times and it looks like things are back to normal at this time. If you experience any delays, please contact support. -Seohee

Previous updates available after the jump →


3:17 am

Web VPS host homie-vserver198 moved to new hardware and restoring from backups (RESOLVED)

Posted (January 21st, 2012 at 3:17 am PST) by jj

Due to issues with the RAID controller on the Web VPS host ‘homie-vserver198′, we have moved the server on to new hardware and are restoring all data from our backup server to the new server. During this process, the data on the backup server will be mounted directly over NFS to allow all services to remain online for the duration of the restoration.

We apologize for the inconvenience caused by this unplanned outage. We will keep this post updated with as much detail as we can provide. If you have any questions or concerns, please contact Support under Support > Contact Support, in your panel.

Please note, this is regarding the Web VPS host machine ‘homie-vserver198’ and no other services (email, MySQL) or servers are affected. To check if you are on a VPS guest from this host, please go to the Users > Manage Users page in your Web Panel and check the “machine” for your user. The affected VPS guests can be found in the first comment below this post.

Update Jan 25th, 1:32pm PST: The restore is complete and all VPS guests are online! If you notice any issues with missing or old data on your account, please be sure to contact support so we can help! -Seohee

Previous updates available after the jump →


9:25 am

Changing Shell/FTP Passwords due to Security Issue (RESOLVED)

Posted (January 20th, 2012 at 9:25 am PST) by Oscar

Last night we detected some unauthorized activity within one of our databases. While we don’t have evidence that customer passwords were taken at this time, we’re forcing a change out of caution. Please login to our web panel and change any passwords you may have with us. We’ll keep this post updated as we get more information.

Update Jan 23rd, 12:16pm PST: We are still actively working through hundreds of support requests due to the delay with password changes in the Web Panel. The great news is that password changes in the Web Panel are now taking less than 10 minutes to complete!

All of our mandatory password changes were done over the weekend, so if you are unable to log in to your FTP/SFTP/Shell accounts at this time, please request another password change and that should resolve your issue!

If you still can’t access your FTP/Shell after that, please contact support with the details, so we can assist you. -Patrick

Previous updates available after the jump →


3:06 am

Shared web server ‘portvila’ moved to new hardware (RESOLVED)

Posted (January 20th, 2012 at 3:06 am PST) by chalise

Due to hardware issues on shared server ‘portvila’, we have moved the server on to new hardware. The data has already been copied over, and after one final sync, sites should be back up and running again. We estimate this will take approximately one hour.

Please note, this is regarding the shared web server ‘portvila’ and no other services (email, MySQL) or servers are affected. To check if you are on this server, please go to the Users > Manage Users page in your Web Panel and check the “machine” for your user.

We apologize for the inconvenience caused by this unplanned outage. We will keep this post updated with as much detail as we can provide. If you have any questions or concerns, please contact Support under Support > Contact Support, in your panel.

Update Jan 20th, 4:05am PST: All sites and services on ‘portvila’ are back up and running normally once again. Thank you for your patience! If you have any further issues, please be sure to contact support. – Chalise.


11:21 am

Intermittent Mail Instability on homiemail-sub5 (RESOLVED)

Posted (January 19th, 2012 at 11:21 am PST) by brians

You are likely experiencing issues while attempting to access your email, send outgoing messages, etc. Our admins are diligently working towards diagnosing and repairing this problem. We will provide an update once more information is made available. We sincerely apologize for the inconvenience this has caused. Thanks again for bearing with us. Your continued patience is greatly appreciated.

To check if you’re affected by this, go to your DreamHost panel and load up your Account Status in the upper right corner. ‘Your Email Server’ should read as homiemail-sub5. No other mail servers or services, such as web or mysql are affected by this problem.

UPDATE Jan 26th 1:50pm PST: After continuous monitoring over the last few days, our administrators have reported no further problems or timeouts. If you run into any issues with your email, please do contact support and let us know. -Oscar Previous updates available after the jump →


10:28 am

Hambone MySQL Downtime Due to RAID Issue (RESOLVED)

Posted (January 19th, 2012 at 10:28 am PST) by Justin@DH

The hambone mysql server is having a RAID card issue, and is not booting up. It is in our LAX datacenter and we have an admin already on his way. This is causing downtime for mysql services for customers using this shared server, which you can find in your control panel by clicking the “Account Status” link on the top right, and looking in the “Your MySQL Server” field. When we have further info, an ETA or a diagnosis we will update this post.

Update Jan 19, 1:00pm PST:The server is back up, but we are planning to move all services off of it to other mysql servers nearby. The hardware is going to be retired. The moves will not be noticable, as the data is copied while the mysql service is running. If you notice further issues please contact support. – Justin

Update Jan 19, 11:30am PST:The admin is still working on the server, but it so far has not accepted the replacement raid card either. Another one is being installed and we are going from there. When we have more info we will update this post again. Sorry for the downtime related to this issue. – Justin


11:12 pm

Emergency network maintenance Thursday, Jan 19th 8pm-12am PST (RESOLVED)

Posted (January 18th, 2012 at 11:12 pm PST) by patrickm

As a follow-up to the network outage we experienced on Wednesday and the maintenance window on Wednesday night, we will be performing more maintenance on the core routers at 8pm-12am PST on Thursday Jan 19th. At that time we will be replacing the bad line card in the core router, re-establishing the links between the two routers and then physically moving half the links back to the other core router. This maintenance should cause about 5-10 minutes of downtime.

Update Jan 19th, 11:50pm PST: Apologies for not updating this post when we began the maintenance, but it went off without a hitch! We are resolving this post now, but please feel free to contact Support with any questions! -Patrick


9:25 pm

Web VPS host ‘homie-vserver38′ moved to new hardware and restoring from backups (RESOLVED)

Posted (January 18th, 2012 at 9:25 pm PST) by Seohee

Due to issues with the RAID controller on the Web VPS host ‘homie-vserver38′, we have moved the server on to new hardware and are restoring all data from our backup server to the new server. During this process, the data on the backup server will be mounted directly over NFS to allow all services to remain online for the duration of the restoration.

We apologize for the inconvenience caused by this unplanned outage. We will keep this post updated with as much detail as we can provide. If you have any questions or concerns, please contact Support under Support > Contact Support, in your panel.

Please note, this is regarding the Web VPS host machine ‘homie-vserver38’ and no other services (email, MySQL) or servers are affected. To check if you are on a VPS guest from this host, please go to the Users > Manage Users page in your Web Panel and check the “machine” for your user. The affected VPS guests can be found in the first comment below this post.

Update Jan 23rd, 4:28pm PST: The RAID card was replaced and the restore is complete! We appreciate your patience while we got this worked out. If you are experiencing any problems at this time, please contact support and let us know. -Seohee

Previous updates available after the jump →


4:37 pm

Emergency network maintenance tonight Jan 18th 8pm-12am (RESOLVED)

Posted (January 18th, 2012 at 4:37 pm PST) by andrea

Due to the network outage that occurred today in one of our datacenters, we’re planning an emergency maintenance to make sure all services are fully operational again. This clean up may cause some connectivity issues, for which we apologize. Please check back for updates as the maintenance progresses. We appreciate your patience and understanding while we check and restore all services affected. – Andrea

Update Jan 18th, 10:15pm PST: Ahead of schedule, we have resolved all outstanding issues in all of our datacenters. The only outstanding case is the reduced routing power, but it shouldn’t be affecting any of your services.
If you’re experiencing any problems with any of your services, please contact support with the details, so we can assist you.

We’ll be updating our original network outage post shortly with a follow up on what caused the downtime. – Andrea

Previous updates available after the jump →


12:46 pm

Major network outage (RESOLVED)

Posted (January 18th, 2012 at 12:46 pm PST) by Oscar

We are currently experiencing a network outage and are investigating the cause.  Our admin team is  working on the issue and we will update this post as soon as we have more information.

Update Jan 18th, 11:03pm PST: We have completed the emergency network maintenance for tonight! We have isolated the cause of the network failure to a bad line card in one of our core routers. The issue manifested itself in the links that connect our two core routers together. At this time we are running everything from a single core router, which has our services fully online!

We are scheduling another maintenance window for tomorrow night to resolve this issue completely! We thank you for your patience throughout this outage! -Patrick

Previous updates available after the jump →


 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
8:42 am

Web VPS host ‘homie-vserver134′ moved to new hardware and restoring from backups (RESOLVED)

Posted (January 18th, 2012 at 8:42 am PST) by Oscar

Due to issues with the RAID controller on the web VPS host ‘homie-vserver134′, we have moved the server on to new hardware and are restoring all data from our backup server to the new hardware. During this process, the data on the backup server will be mounted directly over NFS to allow all services to remain online for the duration of the restoration.

We apologize for the inconvenience and will keep this post updated as further information becomes available. If you have any questions or concerns, please contact Support under Support / Contact Support, in your panel.

Please note, this is regarding the web VPS host machine ‘homie-vserver134’ and no other services (email, MySQL) or servers are affected. To check if you are on a VPS guest from this host, please go to the Users > Manage Users page in your Web Panel and check the “machine” for your user. The affected VPS guests can be found under all the updates in this post.

Update Jan 23rd, 2:03pm PST: This restoration is now complete! The server is completely online and not running from a mounted backup server any longer! All VPS guests are online for this Web VPS host server!

If you notice any issues with the restored data on your sites, such as missing or old files, please contact support for assistance. Otherwise, this post is now being marked as Resolved! -Patrick

Previous updates available after the jump →


8:22 am

Emergency Maintenance with our Easy one-click installs (RESOLVED)

Posted (January 18th, 2012 at 8:22 am PST) by Oscar

Our easy one click installs are currently down while we perform emergency maintenance. If you notice one of your easy one click installs is not resolving or loading, it is because all of the servers for our easy one clicks are currently offline. We will update this post as soon as we have more information.

Update Jan 18th, 11:00am PST: The maintenance is now down and the easy one click servers are back up. Should you have any issues with your easy one click sites, please contact our support staff and we will be glad to look into it for you. -Oscar


5:56 am

Intermittent Webmail Stability Issues (CLOSED)

Posted (January 18th, 2012 at 5:56 am PST) by chalise

Unfortunately, we are again noticing intermittent issues with our webmail service, and our admins are actively investigating this. This currently seems to only be affecting SquirrelMail; access via RoundCube or an email client (ie. Thunderbird, Outlook, Mac Mail.app, etc.) should continue to work normally.

Thanks once again for your patience; we will update this post again with further updates as they become available.

Update Jan 18th, 1:59pm PST: We’re closing this post, as it’s related to the network issues we’re having here:

http://www.dreamhoststatus.com/2012/01/18/major-network-outage-4/

We’ll update that post as we get more details! — Brian H

Previous updates available after the jump →


12:48 am

Shared web server ‘samick’ offline for RAID card replacement

Posted (January 18th, 2012 at 12:48 am PST) by chalise

We apologize for the inconvenience, but we had to take the shared web server ‘samick’ offline briefly for a RAID card replacement.  We expect this maintenance to take approximately 20 minutes.  We will update this post when the issue has been resolved!

Please note, this is regarding the shared web server ‘samick’ and no other services (email, MySQL) or machines are affected. To check if you’re on this server, please go to the DreamHost panel and check the home server for your user on the “Users > Manage Users” page.

Update Jan 18th, 1:15am PST: The server is back up and running without any issues! If you have any questions, please contact Support! – Chalise.


10:38 pm

Web VPS host ‘homie-vserver154′ moved to new hardware and restoring from backups (RESOLVED)

Posted (January 17th, 2012 at 10:38 pm PST) by chalise

Due to issues with the RAID controller on the web VPS host ‘homie-vserver154′, we have moved the server on to new hardware and are restoring all data from our backup server to the new hardware. During this process, the data on the backup server will be mounted directly over NFS to allow all services to remain online for the duration of the restoration.

We apologize for the inconvenience and will keep this post updated as further information becomes available. If you have any questions or concerns, please contact Support under Support / Contact Support, in your panel.

Please note, this is regarding the web VPS host ‘homie-vserver154′ and no other services (email, MySQL) or servers are affected. To check if you are on a VPS guest from this host, please go to the Users > Manage Users page in your Web Panel and check the “machine” for your user. The affected VPS guests can be found in the first comment below this post.

Update Jan 24th, 12:54pm PST: This restore is now complete! The server is no longer mounting backup data and all VPS guests are reporting online. If you find that you have old data or missing data on your VPS, please contact support and we will be happy to assist! -Patrick

Previous updates available after the jump →


6:46 pm

Emergency maintenance scheduled for ‘homiemail’ sub1, sub 3, and sub4 servers tonight @ 7pm PST (RESOLVED)

Posted (January 17th, 2012 at 6:46 pm PST) by Seohee

The ‘homiemail’ sub 1, sub 3, and sub 4 mail servers will be going through network maintenance tonight. There is no expected outage during this time, and is estimated to take about 30 minutes.

To check if you’re affected by this, go to your DreamHost panel and load up your Account Status in the upper right corner. ‘Your Email Server’ should read as homiemail-master, or homiemail-sub3 or homiemail-sub4. Homiemail-sub5 isn’t affected by this maintenance, and neither are other services, such as web or MySql.

If you run into any issues, please contact support.

Update Jan 17th, 7:55pm PST: All maintenance is complete now. If you’re having any trouble with your email then please contact support. Thanks. -Dan J


1:54 pm

Shared web server ‘dobro’ moved to new hardware and restoring from backups (RESOLVED)

Posted (January 17th, 2012 at 1:54 pm PST) by Brandon

Due to issues with the RAID controller on the shared web server ‘dobro’, we have moved the server on to new hardware and are restoring all data from our backup server to the new hardware. During this process, the data on the backup server will be mounted directly over NFS to allow all services to remain online for the duration of the restoration.

We apologize for the inconvenience caused by this unplanned outage. We will keep this post updated with as much detail as we can provide. If you have any questions or concerns, please contact Support under Support > Contact Support, in your panel.

Please note, this is regarding the shared web server ‘dobro’ and no other services (email, MySQL) or servers are affected. To check if you are on this server, please go to the Users > Manage Users page in your Web Panel and check the “machine” for your user.

Update Jan 23rd, 6:08pm PST: The server restore is now complete! If you run into any issues, please contact support! Thanks for your patience! -Seohee

Previous updates available after the jump →


5:28 am

Shared web server ‘tygra’ moved to new hardware and restoring from backups (RESOLVED)

Posted (January 17th, 2012 at 5:28 am PST) by patrickm

Due to issues with the RAID controller on the shared web server ‘tygra’, we have moved the server on to new hardware and are restoring all data from our backup server to the new hardware. During this process, the data on the backup server will be mounted directly over NFS to allow all services to remain online for the duration of the restoration.

We apologize for the inconvenience caused by this unplanned outage. We will keep this post updated with as much detail as we can provide. If you have any questions or concerns, please contact Support under Support > Contact Support, in your Web Panel.

Please note, this is regarding the shared web server ‘tygra’ and no other services (email, MySQL) or servers are affected. To check if you are on this server, please go to the Users > Manage Users page in your Web Panel and check the “machine” for your user.

Update Jan 19th, 2:30pm PST: The restore has finished! If you are missing any data on your sites, please contact our support staff and we can look to see if we can restore the data for you. -Oscar Previous updates available after the jump →


10:14 pm

Maintenance scheduled for MySQL server ‘madal’ on Jan 22nd @ 9pm PST (RESOLVED)

Posted (January 15th, 2012 at 10:14 pm PST) by danwy

We’ve scheduled a window of downtime for the MySQL server ‘madal’ in order to perform some maintenance. Downtime should be limited to 30 minutes or so during this time and will only affect MySQL related services. It will only affect site availability if the site is using a database that is hosted on this server. We will update this post shortly before the maintenance window starts and update it when it’s complete. If you have any questions regarding this, please don’t hesitate to contact support to let us know.

Update Jan 22nd, 9:16pm PST: The maintenance is complete and everything is running well at this time! Please contact support if you experience any issues with the MySQL server ‘madal’. -Seohee

Previous updates available after the jump →


12:31 pm

Shared web server ‘porto-novo’ under DDoS attack (RESOLVED)

Posted (January 14th, 2012 at 12:31 pm PST) by Oscar

Customers on the shared web server ‘porto-novo’ may be experiencing downtime due to an ongoing denial of service attack. Our admin team is looking into it and working on containing and stopping the attack. Our sincere apologies for any inconvenience this is causing.

Please note, this is only affecting ‘porto-novo’. To check if you’re on the affected machine, please go to the DreamHost panel and check the home server for your user.

Update Jan 14th, 2:40pm PST:  The DDoS attack against ‘porto-novo’ has been blocked. The target apache (that is, where the attacked domains are) has been assigned a new IP address to minimize the chance of further attacks. If you’re on  ‘porto-novo’ and use our name servers (ns1.dreamhost.com, ns2.dreamhost.com) for your domain, you won’t have to do anything else, just sit tight until the new DNS propagates. This may take a couple of hours, but your domain should be loading again shortly. However, if you’re using outside name servers for your domain, you will have to contact the provider with the new IP address. You can find your domain’s new IP address by going to the DreamHost panel and clicking on the DNS link under that domain. -Oscar Previous updates available after the jump →


 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
3:34 pm

Intermittent Stability Issues on WebMail (RESOLVED)

Posted (January 13th, 2012 at 3:34 pm PST) by brians

Our admins are currently investigating the stability problems our WebMail service. In the meantime you will experience intermittent timeouts when using Squirrel Mail and RoundCube. Our admins are diligently working towards resolving the issue. We will update this post as information is made available. Sorry for any inconvenience this has caused. Your continued patience is greatly appreciated.

Update Jan 14th, 3:05pm PST: Our admin team is reporting that all webmail issues should now be resolved. We have been checking incoming support for any new cases of webmail timeout issues and there haven’t been any new cases being reported. At this point we feel confident that we have resolved all intermittent stability issues with webmail and will be resolving this status post. If you have any further issues with webmail, please contact our support team and we will be glad to look into it for you.  -Oscar Previous updates available after the jump →


10:30 am

Network Outage affecting one of our Data Centers (RESOLVED)

Posted (January 13th, 2012 at 10:30 am PST) by Oscar

We are currently experiencing a network outage at one of our data centers. For some customers this is affecting access to the your sites as well as the web panel. Our admin team is currently working on the issue and we will update this post as soon as we have more information.

Update Jan 13th, 12:00pm PST:  The transit link causing the issue has been shutdown and all traffic has been routed to our other transit providers. We will continue to work on resolving the issue with the down transit provider, but all other transits are serving traffic without any issues at all. At this time, we’re resolving the status post as all everything is now back up and working. If you are still having issues with your sites being down, please contact our support team and we will be glad to help  you out. -Oscar

Update Jan 13th, 11:40am PST:  We’re still working on the issue, but most customers are reporting that sites are back up for them. We’ll update this post once we get more information from our network engineers. -Oscar

Update Jan 13th, 10:55am PST:  Our network engineer is already on site and working on the issue. One of our transit links  is currently down which is causing the issues. We will continue to update this post as soon as we have more information. -Oscar


4:25 am

Shared web server ‘dodoma’ offline for RAID card replacement (RESOLVED)

Posted (January 13th, 2012 at 4:25 am PST) by patrickm

We apologize for the inconvenience, but we had to take the shared web server ‘dodoma’ offline briefly for a RAID card replacement.  We expect this maintenance to take approximately 30 minutes.  We will update this post when the issue has been resolved!

Please note, this is regarding the shared web server ‘dodoma’ and no other services (email, MySQL) or machines are affected. To check if you’re on this server, please go to the DreamHost panel and check the home server for your user on the “Users > Manage Users” page.

Update Jan 13th, 5:41am PST: Sorry for the delay! The server is now back online and running well! Please have a look and contact Support with any questions! -Patrick


3:35 pm

Scheduled maintenace on Flint and Jabberjaw MySQL servers on Jan 13th @ 10PM PST (RESOLVED)

Posted (January 12th, 2012 at 3:35 pm PST) by danwy

We have rescheduled maintenance on the Flint and Jabberjaw MySQL servers for Friday, January 13th, at 10pm PDT. Our admins need to replace the batteries in these servers and need to take them down to do so. Total downtime shouldn’t be more than 10 or 15 minutes for each server. If you have any questions regarding this maintenance, please contact our support team for further information.

Update Jan 14th, 11:43pm PST: Our apologies for the delay in announcing the resolution of this post. This issue has been resolved since 2am and it all went through without any problems. If you run into any problems with one of these machines please let us know. -JJ G


3:10 pm

Rescheduled maintenance of MySQL machines on Jan 14th @10pm PST (RESOLVED)

Posted (January 12th, 2012 at 3:10 pm PST) by danwy

Last week we posted about some scheduled maintenance for several of our MySQL machines. Here is the post in question:

http://www.dreamhoststatus.com/2012/01/06/performing-scheduled-maintenace-on-mysql-machines-on-jan-7th-10pm-pdt

The following machines were scheduled to be upgraded:

tramp
barbaro
bang
vuvuzela
babelfish
courage
trombone
smelt

We were only able to complete the upgrades on tramp, vuvuzela, and courage, so we will be upgrading the rest of them this coming Saturday night, January 14th, at 10pm PST. Each machine should only be down for 10 or 15 minutes while we replace the batteries in the machines above. If you have any questions or concerns about this maintenance, please contact our support team for further assistance.

Update Jan 15th, 4:30am PST: Our admins have reported that maintenance for this issue has been completed successfully and we are resolving this post. If anyone has problems with MySQL on these machines please let support know. -JJ G Previous updates available after the jump →


7:22 am

Shared web server “yangon” down due to hardware issues (RESOLVED)

Posted (January 12th, 2012 at 7:22 am PST) by Oscar

The shared web hosting server “yangon” is currently down due to issues with it’s hardware. Our admin team is working on the issue and we will update this post as soon as we have an update for you.

Please note, this is regarding the web server ‘Godin’ and no other services (email, MySQL) or machines are affected. To check if you’re on this server, please go to the DreamHost panel and check the home server for your user on the “Users > Manage Users” page.

Update Jan 12th, 7:35am PST: The server is now back up and working without any issues!


3:22 pm

Shared web server ‘hachiman’ offline due to unscheduled RAID card replacement (RESOLVED)

Posted (January 11th, 2012 at 3:22 pm PST) by Brandon

We are currently working on replacing the RAID controller in the shared web server ‘hachiman’.  We expect to have this server back online shortly and we will provide any updates we can!

Please note, this is regarding the shared web server ‘hachiman’ and no other services (email, MySQL) or machines are affected. To check if you’re on this server, please go to the DreamHost panel and check the home server for your user. -Brandon

Update Jan 12th, 5:30am PST: This server has been back online for a couple hours, as we were able to get the old hardware working again! Please contact Support if you are still experiencing any issues! -Patrick

Previous updates available after the jump →


10:30 am

Shared mail service issues on homiemail-sub5 is being investigated (RESOLVED)

Posted (January 11th, 2012 at 10:30 am PST) by andrea

We added several new machines to the homiemail-sub5 system in the past 10 days. Unfortunately, adding the extra machines didn’t resolve all the issues that we expected to be gone by now, so we are currently investigating other causes. Scripts are running to test sending, receiving and connections to and from the machines, as from your messages that’s what seems to be the problem.

Just a quick tip that may help: many customers reported that they are able to successfully use webmail just not other email clients, so you may want to give that a try in the interim.

Very sorry about the inconvenience this has been and still causing. We’ll post periodic updates or as soon as the situation changes.

To check if you’re affected by this, go to your DreamHost panel and load up your Account Status in the upper right corner. ‘Your Email Server’ should read as homiemail-sub5. No other mail servers or services, such as web or mysql are affected by this problem. – Andrea

Update Jan 18th, 11:50am PST: We are closing out this status post now, as there have been no additional issues of this being a problem in the last 24 hours. If you are having any further issues with mail, please check this status page for other status posts regarding mail problems or contact support for further assistance with the matter. -dan w

Previous updates available after the jump →


6:27 pm

Network Maintenance Tuesday Night, January 10th, 11pm – 6am PST (RESOLVED)

Posted (January 10th, 2012 at 6:27 pm PST) by Dan J

We’ll be performing some emergency network maintenance tonight January 10th, between 11pm and 6am PST. There may be some intermittent networking issues for connections between two of our datacenters during this maintenance. The vast majority of customers should see no issues as the connections from each individual datacenter out of our network will remain online.  Both our Senior Network Engineers will be up and monitoring the situation in addition to our normally scheduled admin and datacenter team.

Please contact support should you experience any unusual issues such as sites not being able to connect to the database.

Update Jan 11th, 8:20am PST: The maintenance was completed without any issues earlier this morning and there were no major outages that came from this maintenance window. Should you have any issues with your sites please contact our support team and we will be glad to look into for you. -Oscar


2:15 pm

Shared web server ‘manama’ being moved to new hardware (RESOLVED)

Posted (January 10th, 2012 at 2:15 pm PST) by Dan J

We apologize for the short notice, as the shared web server, ‘manama’ was taken offline to complete an unscheduled file-system integrity check. As that may take a while to complete we are going to move the server to new hardware. The new hardware will be serving data from our backup server while we copy data to the new servers local hard drive.

Please note, this is regarding the shared web server ‘manama’ and no other services (email, MySQL) or machines are affected. To check if you’re on this server, please go to the DreamHost panel and check the home server for your user.

Update Jan 12th, 5:58am PST: Our apologies for not providing another update last night. This server has been up and running again for about 13 hours.  Please contact Support if you notice any issues! -Patrick

Previous updates available after the jump →


 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
4:41 am

WebFTP outage (RESOLVED)

Posted (January 10th, 2012 at 4:41 am PST) by patrickm

We are currently experiencing an outage with our WebFTP service. Our Admins are working on resolving the error and we hope to have this service up and running again shortly! We will update this post with any details we can provide!

Note: This issue is affecting only WebFTP (http://webftp.dreamhost.com). Regular FTP through your FTP software client will still function normally. Please contact Support with any questions!

Update Jan 10th, 5:11am PST: This issue is now completely resolved! Thanks for your patience as we got to the bottom of this. Please contact Support if you continue to have any issues with WebFTP! -Patrick


5:49 pm

Panel down (RESOLVED)

Posted (January 9th, 2012 at 5:49 pm PST) by Seohee

The DreamHost control panel (https://panel.dreamhost.com) is currently down. Our administrators are looking into the issue now, and we’ll make sure to update this as soon as we have further details or a resolution.  We appreciate your patience.

Update Jan 9th, 6:26pm PST: The panel is back up and running smoothly. It appears to have been caused by a problem with an internal database. If you run into any further issues, please contact support and let us know! Thanks again for your patience while we got this worked out. -Seohee


1:25 pm

Shared machine ‘ Godin ‘ is moving to new hardware (RESOLVED)

Posted (January 9th, 2012 at 1:25 pm PST) by brians

The shared web hosting server Godin had to be taken offline for an unscheduled file-system check. The file-system check failed though, so we are now moving this server over to new hardware. The new machine will mount our backup server to serve website files while we copy files over to the new machine.

Please note, this is regarding the web server ‘Godin’ and no other services (email, MySQL) or machines are affected. To check if you’re on this server, please go to the DreamHost panel and check the home server for your user on the “Users > Manage Users” page.

Update Jan 15th, 4:50pm PST: The restore to the new hardware is now complete, so we’re resolving this status post. If you are having issues with your web server, please contact support with the details, so we can assist you, as it’s likely unrelated. – Andrea

Previous updates available after the jump →


6:26 pm

Web VPS host homie-vserver190 being moved to new hardware (RESOLVED)

Posted (January 7th, 2012 at 6:26 pm PST) by Seohee

The web VPS host homie-vserver190 is being moved to new hardware due to an issue with its filesystem and one of its partitions. This will cause some downtime for the VPS on this host until the failover (move) has begun. Once it has, they should come back up and run properly (which will be fairly soon). We apologize for the inconvenience that this situation causes. We’ll make sure to keep this post updated as we receive further information on the progress of the move.

Please contact Support from the “Support > Contact Support” section of your Web Panel if you have any questions!

Please note, this is regarding the VPS host machine ‘homie-vserver190’ and no other services (email, MySQL) or machines are affected. To check if you’re on a VPS on this host, please go to the DreamHost panel and check the home server for your user. The affected VPS guests can be found in the first comment below this post.

Update Jan 15th, 8:30pm PST: The data move to the new machine is now complete. Thank you for your patience during this process. If you have any problems related to your VPS, please contact support with the details, as it’s likely unrelated to this case. – Andrea

Previous updates available after the jump →


1:46 am

Shared web server ‘mcgurn’ down for unscheduled file system check (RESOLVED)

Posted (January 7th, 2012 at 1:46 am PST) by jj

The shared web hosting server mcgurn has been taken offline for an unscheduled file-system check. This check is being performed to verify the integrity of the data on the server. We will update this post as this process continues.

Please note, this is regarding the web server ‘mcgurn’ and no other services (email, MySQL) or machines are affected. To check if you’re on this server, please go to the DreamHost panel and check the home server for your user on the “Users > Manage Users” page.

Update Jan 7th, 10:25am PDT: This server should be back online, and running off of the original hardware. If you’re seeing any problems, please let us know via the support system ASAP! — Brian H

Update Jan 7th, 5:30am PDT:  Services remain stable on the server currently serving the sites. It seems that the file system check has completed on the original hardware and we’ll post an update on what is going to happen next as soon as that is known to us. -JJ G Previous updates available after the jump →


7:45 pm

web server “tabbuts” is currently down. (RESOLVED)

Posted (January 6th, 2012 at 7:45 pm PST) by Brian@dreamhost

Currently, the web server “tabbuts” is down due to problems with its file system. We’re in the process of investigating this, and hope to have service restored on it shortly. We will update this as things happen. — Brian H

Update Jan 6th, 8:10pm PDT: The server is backup and working now! Sorry about any trouble this caused! — Brian H


1:23 pm

Performing scheduled maintenace on mysql machines on Jan 7th @ 10PM PDT (RESCHEDULED)

Posted (January 6th, 2012 at 1:23 pm PST) by brians

Our admins will replace the batteries in machines listed below. We anticipate 15 minutes of downtime. Any mysql driven site ( wordpress, phpbb, etc ) will not be accessible at this time. We will update this with once the battery swap is complete. Sorry for any inconvenience this has caused. Your continued patience is greatly appreciated.

tramp
barbaro
bang
vuvuzela
babelfish
courage
trombone
smelt

If you wish to confirm whether you are affected by this outage, please log into our Web Panel and click the ” Account Status ” link. It’s located in the upper-right section of our panel interface.

Update Jan 7th, 2:30am PDT: The issue with courage has been resolved. Maintenance on on the remaining servers has been delayed and will be rescheduled for another time. -JJ G Previous updates available after the jump →


1:13 pm

Performing scheduled maintenace on Flint and Jabberjaw on Jan 6th @ 10PM PDT (RESOLVED)

Posted (January 6th, 2012 at 1:13 pm PST) by brians

These are 2 shared mysql machines. Our admins will replace the batteries in both machines. We anticipate 15 minutes of downtime. Any mysql driven site ( wordpress, phpbb, etc ) will not be accessible at this time. We will update this with once the battery swap is complete. Sorry for any inconvenience this has caused. Your continued patience is greatly appreciated.

If you wish to confirm whether you are affected by this outage, please log into our Web Panel and click the ” Account Status ” link. It’s located in the upper-right section of our panel interface.

Update Jan 7th, 12:20am PDT: Our admins have informed us that maintenance has been completed successfully and everything has been resolved since 11:30pm. -JJ G

Update Jan 12th, 3:30pm PST: There was a miscommunication regarding this upgrade and these two servers actually didn’t get the maintenance done on them yet. We’re rescheduling this maintenance for this Friday, January 13th @ 10pm PST. We’ll create a new status post for this rescheduled maintenance window.


9:22 pm

Shared web server ‘miaplacidus’ down (RESOLVED)

Posted (January 5th, 2012 at 9:22 pm PST) by Seohee

The shared web server ‘miaplacidus’ is currently down due to issues with mounting its partitions at boot. Our system administrators are investigating the problem now, and we’ll make sure to post an update as soon as we have further information or a resolution.

Please note, this is regarding the shared web server ‘miaplacidus’ and no other services (email, MySQL) or machines are affected. To check if you’re on this server, please go to the DreamHost panel and check the home server for your user.

Update Jan 5th, 10:45pm PDT: We’ve been told that the error has been corrected. We’ve been watching the machine and everything has remained stable, so this issue has now been resolved. If you have any trouble with a site on this machine please let support know. -JJ G Previous updates available after the jump →


6:30 pm

Improved version of customer surveys!

Posted (January 5th, 2012 at 6:30 pm PST) by andrea

You may have already received one of these, but if you haven’t: we made some improvements on the customer surveys we send out in random support replies. We take customer satisfaction seriously, so we decided to collect more specific and extensive information for benchmarking purposes. We would like to learn more about what we’re doing right or where we can improve to ensure even better service for you. We appreciate your participation and welcome your input. You can also opt-out of the surveys right in the email. As always, don’t hesitate to ask if you have any questions or concerns.


 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
1:21 pm

Web VPS host homie-vserver174 is being moved to new hardware (RESOLVED)

Posted (January 5th, 2012 at 1:21 pm PST) by Brandon

Initially we posted and informed you that a filesystem check needed to be performed. This process has taken longer than expected. Our admin felt that the best course action would be to move the machine to new hardware. A NFS mount to the backup machine ( where your backup data is stored ) will be created. This will allow your sites to function as the filesystem check is being performed on the old hardware. I apologize in advance if the content is a bit out of date. We will continue to update this post as information is made available. Please check back.

Please note, this is regarding the VPS host machine ‘homie-vserver174’ and no other services (email, MySQL) or machines are affected. To check if you’re on this server, please go to the DreamHost panel and check the home server for your user.

The affected VPS guests can be found in the first comment below this post.

Update Jan 18th, 7:14pm PST: The restore is now complete and all VPS guests should be up and running well! If you experience any issues, please contact support. We appreciate your patience! -Seohee Previous updates available after the jump →


11:55 pm

Performing maintenance on the machine that stores DEDICATED backups on Jan 5th at 11:00am PST (RESOLVED)

Posted (January 4th, 2012 at 11:55 pm PST) by jj

Our admins will be upgrading the machine that stores the backups. Your dedicated machine mounts to the backup machine. Therefore you will notice an inflated load averages on your machine. Your machines will remain online during the maintenance. However, you may experience a noticeable lag while navigating through your website.

The maintenance will be completed by noon. So you are looking at a hour of instability. We sincerely apologize if this poses any problems for you. Thanks for your understanding. Please check back for updates.

Also, I’ve provide a list of the machines that will be affected. It’s located in the comments section of this post.

Update Jan 5th, 3:17pm PST:The maintenance has successfully completed! Should you be seeing any issues on your dedicated server, please contact support. -Brandon

Previous updates available after the jump →


5:14 pm

If you received a message from Email Performance Improvements Team…

Posted (January 4th, 2012 at 5:14 pm PST) by andrea

If you’re one of those customers who received a message from the ‘Email Performance Improvements Team’, please don’t fret. Those messages went out by mistake. You may safely ignore and delete them, there is no need to reply or worry about your email address. We promise!
You may be receiving a different message related to your email address in the near future, with more information, so keep a lookout for that.

Sorry about the confusion! – Andrea


3:22 pm

Homiemail-sub5 mail service expansion (RESOLVED)

Posted (January 4th, 2012 at 3:22 pm PST) by andrea

During peak hours, some of our customers using homiemail-sub5 for their mail service have been experiencing delay in getting mail or problems with accessing or sending mail. This problem is apparent regardless of which email clients are used. We are aware of this problem and are working on remedying it by adding more servers to that mail system. The servers are ready to go, and they should provide plenty more room for processing. We expect relief from this issue by no later than the end of this week.

To check if you’re affected by this, go to your DreamHost panel and load up your Account Status. ‘Your Email Server’ should read as homiemail-sub5. No other mail servers or services, such as web or mysql are affected by this problem.
If you’re on a different email server and are having issues using Webmail, you may be (or have been) affected by our Webmail problems. If it’s a different client, then please contact support with the details for assistance.

We apologize for the inconvenience and will update this post as soon as we have new information. – Andrea

Update Jan 10th, 4:40pm PDT: We should have the remaining issues wrapped up on this mail cluster. If you’re still seeing problems, please let our support staff know right away! — Brian H

Previous updates available after the jump →


3:06 pm

Webmail issues being investigated (RESOLVED)

Posted (January 4th, 2012 at 3:06 pm PST) by andrea

For the past few days, we’ve been having some problems with our webmail system that caused login and other access problems for our customers using Squirrel Mail or RoundCube. We are aware of the issues and our system administration team is investigating and looking for a permanent solution.

In the meantime, it may be advised that you rely on other email clients to access your mail. We understand that’s not ideal, but it’s an alternative. We apologize for all inconvenience this has been causing. We’ll be posting updates as soon as we have more information about the resolution.

Please note, this post is regarding Webmail usage, no other services like regular email client access/sending/receiving, web or mysql are affected. If you’re having other email client issues or delayed mail, you may be affected by another status case: Homiemail-sub5 mail service expansion . – Andrea

Update Jan 8th, 11:58pm PST: Apologies for not resolving this post any sooner! The issues we were experiencing with WebMail are resolved and the service is working normally again. If you notice any issues, please contact Support. -Patrick

Previous updates available after the jump →


11:29 am

Webserver VPS “homie-vserver150″ offline for unscheduled RAID card replacement ( RESOLVED )

Posted (January 4th, 2012 at 11:29 am PST) by Justin@DH

The webserver VPS host machine “homie-vserver150″, is having stability issues currently due to its RAID card, and it is in need of replacement. Our data center staff are en route to get this card replaced, which we believe will fix the stability issues. This is affecting web services only, so email, mysql and other services will continue to function normally. When we have any further information or ETA, we will update this post.

This issue affects only the following webserver VPS:

ps23615 ps23992 ps24245 ps24324 ps24753 ps24853 ps24924 ps24971 ps24987 ps25148 ps25272 ps25387 ps25414 ps25464 ps25567 ps25752 ps25782 ps26504 ps28749 ps28885 ps29179 ps29302 ps29347 ps29451 ps29687 ps29689 ps29770 ps29932 ps29960 ps30418 ps30780 ps30787 ps30800 ps30847 ps30851 ps31117 ps34996 ps35471 ps36923 ps42228 ps44130 ps44592 ps47027 ps48039 ps48051 ps48053 ps58256

Update Jan 4th, 1:30pm PDT: The RAID card has been replaced and all VPS guests are back online. Sorry for any inconvenience this has caused. Your patience is greatly appreciated. If you are still having problems, please contact Tech Support. We will gladly take a look at your specific issue. – Brian S


8:53 am

WordPress 3.3.1 is now available

Posted (January 4th, 2012 at 8:53 am PST) by brians

You can manually upgrade to this version through the Goodies::One-Click Installs section of our Panel https://panel.dreamhost.com/index.cgi?tree=goodies.installer&. Of course version 3.3.1 will be setup on all brand new installs as well.

For the users who configured your install to ” Upgrade Automatically ” do not have to do anything. Since the software upgrade will be pushed out to your installs momentarily.

You can learn more about this release here:

http://wordpress.org/news/2012/01/wordpress-3-3-1/


2:49 pm

Email Relaying Delay (RESOLVED)

Posted (January 3rd, 2012 at 2:49 pm PST) by Oscar

Mail is backed up on some of our relay machines, causing some mail delivery to be delayed. Our admin team is currently working on cleaning these queues up and getting mail flowing again. We’ll update this post when things are back to normal.

Update Jan 3rd, 4:06pm PDT: Mail should be flowing normally now. The root cause appears to be due to an intrusion detection system that ran into trouble. Servers will be monitored overnight for stability by our administrators. If you see that any issues persist, please contact Support and let us know. We sincerely appreciate your patience while we got this worked out. -Seohee


10:24 am

Ongoing networking outage in one of our facilities (RESOLVED)

Posted (January 3rd, 2012 at 10:24 am PST) by brians

We’ve been having networking issues since this morning in our Downtown Los Angeles facility. It has caused three of our DNS servers to not being able to function correctly, aggravating the situation. We rebooted those machines, but the networking outage is still affecting some of our central services, such as access to the DreamHost panel, email and DNS.

Sorry for any inconvenience this has caused. Your continued patience is greatly appreciated. Please check back for updates.

Update Jan 3rd, 6:10pm PDT: We’ve received word from our administrators that this is now resolved. If you run into any issues you believe may be related, please do contact Support! -Seohee

Previous updates available after the jump →


12:26 pm

WebMail ( Squirrel Mail ) minor outage (RESOLVED)

Posted (January 2nd, 2012 at 12:26 pm PST) by brians

We had some issues on the central database that hosts WebMail. Our admins have tried remote reboots, but eventually manual reboot was needed. Sorry for all mail issues that are being encountered today. Your patience and understanding is greatly appreciated.

Also, this outage only affects web-based access http://webmail.yourdomain.com. You can still access your mailbox via mail clients, such as Outlook, Mozilla Thunderbird, etc. We will post more details as they are made available. Please check back for updates.

Update Jan 2nd, 4:05pm PST: The cause of the webmail outage is now fixed, and have been for a while, but we needed to monitor it to make sure we don’t see new cases. If you’re still having issues with your mail, it’s likely unrelated. Please contact support with the specifics, and we’ll be happy to help. – Andrea

Previous updates available after the jump →


 
© 1996-2012, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
6:43 am

Messages originating from Webservers are being delayed (RESOLVED)

Posted (January 2nd, 2012 at 6:43 am PST) by brians

Again, this only impacts messages that are sent from webservers. All message generated by your website via sendmail, php mail() will be delayed. Our administration team is currently investigating the issue. We sincerely apologize for the inconvenience this has caused and appreciate your continued patience. Please check back for future updates.

There is an alternative to sending via phpmail, sendamil, etc. Various software programs give you the capability to send via SASL or AUTH. By enabling that option, the message will get delivered from our SMTP servers instead. Therefor you will be bypassing the delayed / queued machine that handles mail sent via phpmail, sendmail, etc. Hopefully this helps to ease the pain for some of you.

Update Jan 3rd, 12:25pm PST: The relay machine is now back in working order, processing mail. There is a very small of a backlock, so you shouldn’t experience more than a minute delay and soon none at all. If you’re still having issues sending mail from your web server or web application, please contact support as it’s likely unrelated. – Andrea

Previous updates available after the jump →


12:08 am

Shared web server “crater” moving to new hardware (RESOLVED)

Posted (January 2nd, 2012 at 12:08 am PST) by Brandon

The shared webserver ‘crater’ was offline earlier due to issues with its filesystem. We tried replacing the RAID card, but we continued to see filesystem errors. Our admins felt that the best course of action was to perform a filesystem check. That check just finished, but in the meantime, move to the new hardware has started.

Please note, this is regarding the shared web server ‘crater’ and no other services (email, MySQL) or machines are affected. To check if you’re on this server, please go to the DreamHost panel and check the home server for your user.

We will update this post with further information as it becomes available, thank you for your patience in this matter.

Update Jan 7th, 7:40am PST: Our apologies for the lack of updates on this issue. This server finished restoring awhile back and the machine had remained stable since it completed, this case should have been marked as resolved earlier. If you are having any trouble with a site on this machine please let support know. -JJ G Previous updates available after the jump →


1:57 am

Web VPS host homie-vserver140 offline for unscheduled file-system check (RESOLVED)

Posted (December 31st, 2011 at 1:57 am PST) by jj

The web VPS host ‘homie-vserver140′ is offline for an unscheduled file-system integrity check. This is required due to issues with the RAID on the server switching to read-only mode following a reboot of the server. We apologize for the inconvenience that this situation causes! Please contact Support from the “Support > Contact Support” section of your Web Panel if you have any questions!

Please note, this is regarding the VPS host machine ‘homie-vserver140’ and no other services (email, MySQL) or machines are affected. To check if you’re on this server, please go to the DreamHost panel and check the home server for your user.

The affected VPS guests can be found in the first comment below this post.

Update Dec 31st, 4:21pm PST: The host should now be backup on the old hardware and all guests should be completely up and running. If you’re seeing problems, please let our support staff know asap! — Brian H

Previous updates available after the jump →


7:40 am

DreamHost Panel offline (RESOLVED)

Posted (December 30th, 2011 at 7:40 am PST) by Brandon

The DreamHost panel ( http://panel.dreamhost.com ) is currently offline due to further networking issues. Our system administrators are actively diagnosing the cause of the issue and hope to have everything resolved shortly. We’ll keep this post up to date as new information becomes available, thank you for your patience in this matter.

 

Update Dec 30th, 8:55am PST: The panel has been back online for some time now, and we’re considering the issue resolved. If we run into any other trouble we’ll reopen this post. -Brandon


7:32 am

Web VPS host homie-vserver175 offline for unscheduled file-system check ( RESOLVED )

Posted (December 30th, 2011 at 7:32 am PST) by Brandon

We apologize for the short notice, as the web VPS host machine, ‘homie-vserver175’ has just been taken offline to complete a file-system integrity check.   A previous File System Check was scheduled on December 26th, however it was not completed as it was taken offline to replace its RAID card. The machine has again gone read-only (5 days later), indicating File System issues which will need to be resolved by running another FSCK to completion in order to prevent data corruption.

Please note, this is regarding the VPS host machine ‘homie-vserver175’ and no other services (email, MySQL) or machines are affected. To check if you’re on this server, please go to the DreamHost panel and check the home server for your user.

The affected VPS guests can be found in the first comment below this post.

Update Dec 30th, 2:40pm PST: The filesytem check has completed. The machine is back online. All services are fully functional. Again, we apologize for any inconvenience this has caused. If you are still having problems, please contact Tech Support. Your issue is likely not related to this status post. -Brian S

Previous updates available after the jump →


4:23 am

Shared web server ‘alcyone’ restoring from backups on new hardware ( RESOLVED )

Posted (December 30th, 2011 at 4:23 am PST) by Brandon

The shared webserver ‘alcyone’ has been experiencing hardware issues and has been moved to new hardware, and is now restoring from backups. Sites on the server will be running from backups, and should all be online shortly.

Please note, this is regarding the shared web server ‘alcyone’ and no other services (email, MySQL) or machines are affected. To check if you’re on this server, please go to the DreamHost panel and check the home server for your user.

We will continue to update this post throughout the day as information becomes available. Please contact support if you experience further issues. Thank you for your patience in this matter.

Update Jan 2nd, 11:47am PST: The restoration has finished. Thanks again for your patience. If you are still having problems, please contact Tech Supports. It is likely that your issue is not related to this status post. -Brian S

Update Jan 2nd, 10:18am PST: The restoration is almost finished. Unfortunately we can no longer provide the completion percentage. We’ll update this post again as further information becomes available. -Brian S

Update Jan 1st, 8:15pm PST: The restoration has now reached 90%, and is progressing without issue. We’ll update this post again as further information becomes available.

Update Jan 1st, 2:40pm PST:  The restore has reached 84.7% and is continuing to go smoothly. We’ll update again soon with another progress update. – Justin

Update Dec 31st, 6:25pm PST: We’ve reached the 70% point for this restore! The next update will come at 80%. — JJ G

Update Dec 31st, 6:25pm PST:  Ok, now we’re at 60% and we’re still chugging along! We’ll update this again as things progress to 70%! — Brian H

Update Dec 31st, 1:25pm PST:  We’re now at 50% restored on this machine. We’ll update this again when the restore is at 60%! — Brian H

Update Dec 31st, 8:00am PST: The restore process continues smoothly and it is now just over 40% complete. We will continue to post updates as the process continues to run. — JJ G

Update Dec 31st, 2:30am PST: The restore process has continued and it is now at roughly 30% complete. We will continue to post updates as new information becomes available. — JJ G

Update Dec 30th, 10:47pm PST: Due to a problem internally, the restore process stalled for a bit. It’s now back up and running, and at 23%. We apologize for the delay! — Brian H

Update Dec 30th, 1:37pm PST: Sites are still up and running on the machine, and the restore progress is now nearing 22%. We’ll keep this post up to date with new information when it becomes available. -Brian S

Update Dec 30th, 7:43am PST: Sites are still up and running on the machine, and the restore progress is now nearing 10%. We’ll keep this post up to date with new information when it becomes available. -Brandon

Update Dec 30th, 5:00am PST: All Apache services have started, and all sites should be up and running. The restoration process is ongoing, and we’ll update this post with further information as it becomes available.  -Brandon


10:59 pm

Shared web server ‘betria’ offline due to unscheduled RAID card (RESOLVED)

Posted (December 29th, 2011 at 10:59 pm PST) by Seohee

We are currently working on replacing the RAID controller in the shared web server ‘betria’.  We expect to have this server back online shortly and we will provide any updates we can!

Please note, this is regarding the shared web server ‘betria’ and no other services (email, MySQL) or machines are affected. To check if you’re on this server, please go to the DreamHost panel and check the home server for your user.

 

Update Dec 29th, 11:57pm PST: The new RAID card is in and the machine is back up and running! Sites should all be back online at this time, but if you’re still experiencing any issues please contact support. -Brandon


7:19 pm

‘Manage Account’ area of Panel down (RESOLVED)

Posted (December 29th, 2011 at 7:19 pm PST) by Seohee

We just had a panel outage due to networking issues with our central databases. The details of that post can be found here:

http://www.dreamhoststatus.com/2011/12/29/panel-down-3/

While the panel is up and running again, the ‘Manage Accounts’ section is unfortunately still experiencing issues. This means that adding or removing hosting plans and canceling your account are not possible at this time. We believe sign-up may also be affected by this outage. Our system administrators are looking into the problem now. We will make sure to update this post as soon as we have further details and/or a resolution. We apologize for the inconvenience.

Update Dec 29th, 8:58pm PST:  The panel is now completely back up and functioning properly. Our admin team is currently restoring the usage data so that the panel usage reports display them properly. If you notice any errors or oddities in the panel, please do contact support and let us know. -Seohee

Previous updates available after the jump →


5:57 pm

Panel down (RESOLVED)

Posted (December 29th, 2011 at 5:57 pm PST) by Seohee

Our panel (https://panel.dreamhost.com) is currently down due to networking issues. Our system administrators are looking into the issue now. We’ll make sure to update this post as soon as we have further details. We appreciate your patience.

Update Dec 29th, 6:47pm PST: Our apologies for the delayed resolution to this post. We had to make sure to get all the details. The issue was due to a network problem with the central database servers. This has been fixed up by our administrators (at around 6:17pm), the panel is once again up and running, and they’re doing all they can to ensure the problem does not arise again. If you run into any further issues with the panel, please  contact support! -Seohee


3:48 pm

Performing maintenance on the machine that stores DEDICATED backups on Dec 30th at 11:00am PST ( POSTPONED )

Posted (December 29th, 2011 at 3:48 pm PST) by brians

Our admins will be upgrading the machine that stores the backups. Your dedicated machine mounts to the backup machine. Therefor you will notice an inflated load averages on your machine. Your machines will remain online during the maintenance. However, you may experience a noticeable lag while navigating through your website.

The maintenance will be completed by noon. So you are looking at a hour of instability. We sincerely apologize if this poses any problems for you. Thanks for your understanding. Please check back for updates.

Also, I’ve provide a list of the machines that will be affected. It’s located in the comments section of this post

UPDATE Dec 30th 11:23am PST: The scheduled maintenance has been postponed until further notice. This status post will be updated when we decide to proceed with the upgrade. Please check back for updates. -Brian S


 
© 1996-2011, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
12:55 pm

Intermittent downtime on shared machine ‘Wellington’ due to RAID issues ( RESOLVED )

Posted (December 29th, 2011 at 12:55 pm PST) by brians

The machine has been rebooted a few times today. Our admins believe that the RAID card on the machine needs to be replaced. A member of the Admin team is en route to the datacenter. In the meantime you will continue to notice downtime. Only web services ( HTTP, FTP, SSH, etc ) are impacted by this outage. We sincerely apologize for any inconvenience this has caused and appreciate your continued patience. Please check back for updates.

UPDATE Dec 29th 2:37pm PST: The faulty RAID card has been replaced. Everything is back to normal. Sorry for any problems that this may have caused. If you are still having problems, please contact Tech Support. It’s likely that your issue is not related to this status post.


3:22 am

Shared web server ‘chamaeleon’ offline for unscheduled RAID card replacement (RESOLVED)

Posted (December 29th, 2011 at 3:22 am PST) by patrickm

We are currently working on replacing the RAID controller in the shared web server ‘chamaeleon’.  We expect to have this server back online shortly and we will provide any updates we can!

Please note, this is regarding the shared web server ‘chamaeleon’ and no other services (email, MySQL) or machines are affected. To check if you’re on this server, please go to the DreamHost panel and check the home server for your user.

Update Dec 29th, 5:00am PST: This server has been online for a little more than an hour following a successful replacement of the RAID controller! Please contact Support if you have any  issues with your hosting on this server! -Patrick


9:48 pm

Irvine datacenter networking issues (RESOLVED)

Posted (December 28th, 2011 at 9:48 pm PST) by Seohee

We are currently experiencing a networking issue in our Irvine datacenter that’s causing downtime for certain dedicated and shared machines, and VPS. Our administrators are looking into the cause of the issue at this time. We will make sure to post an update as soon as we have more information. We appreciate your patience. -Seohee

Update Dec 28th, 10:10pmm PST:We managed to get everything back in order, sorry about the delay. Some customers were up again in as little as five minutes, but some experienced longer delays until around 9:45pm PST. We delayed the status update to make sure everything is running again as expected.
Our system administrators are working on making sure that what caused tonight’s outage is thoroughly looked into and will not be attempted again without making sure it will not have such effect.
If you’re still having issues it’s likely unrelated, so please contact support with the details, so we can assist you. – Andrea


9:31 am

Networking issues affecting a row of machines at our LAX datacenter ( RESOLVED )

Posted (December 26th, 2011 at 9:31 am PST) by brians

Our admins are currently investing a network issue at our LAX datacenter. They are diligently working towards diagnosing the root cause of the problem. We will post additional information as it is made available. We sincerely apologize for any inconvenience this has caused and appreciate your continued patience.

Also, the comments section of this contains a list of the impacted machines.

UPDATE Dec 26th 9:47am PST: Everything has been resolved. Our admins had to clear the arp cache on a few switches. Upon doing so, traffic flow was back to normal. We sincerely apologize for any inconvenience this has caused and appreciate your continued patience. If you are still experiencing problems, please contact tech support with the details.


1:21 am

Web VPS host homie-vserver175 offline for unscheduled file-system check ( RESOLVED )

Posted (December 26th, 2011 at 1:21 am PST) by chalise

We apologize for the short notice, as the web VPS host machine, ‘homie-vserver175’ has just been taken offline to complete a file-system integrity check. The server was rebooted to resolve an issue with the file-system switching to read-only and this check needs to be completed to verify the integrity of the RAID array. We will post more details about this as the check progresses!

Please note, this is regarding the VPS host machine ‘homie-vserver175’ and no other services (email, MySQL) or machines are affected. To check if you’re on this server, please go to the DreamHost panel and check the home server for your user.

The affected VPS guests can be found in the first comment below this post.

Update Dec 26th, 9:10am PST: The file-system check has completed. Our admins replaced the raid card on the machine as well. The host machine and vps guests are back online. We sincerely apologize for any inconvenience this has caused and appreciate your patience. If you are still experiencing downtime, please submit a support request. -Brian S

Previous updates available after the jump →


12:38 pm

Shared web server ‘minkar’ was under DDoS attack, one apache getting new IP (RESOLVED)

Posted (December 25th, 2011 at 12:38 pm PST) by andrea

Customers on the shared web server minkar may be experiencing downtime due to an ongoing denial of service attack since this morning. Our techs are looking into it and working on containing and stopping the attack. Very sorry about the inconvenience this is causing.

Please note, this is also causing issues on machines that are in the same location, namely minelava. To check if you’re on the affected machines, please go to the DreamHost panel and check the home server for your user. – Andrea

Update Dec 25th, 1:05pm PST: The DDoS attack against minkar has been stopped. The target apache (that is, where the attacked domains are) is getting a new IP address to minimize the chance of further attacks. If you’re on minkar and use our name servers (ns1.dreamhost.com, ns2.dreamhost.com) for your domain, you won’t have to do anything else, just sit tight until the new DNS propagates. This may take a couple of hours, but your domain should be loading again shortly. However, if you’re using outside name servers for your domain, you will have to contact the provider with the new IP address. You can find your domain’s new IP address by going to the DreamHost panel and clicking on the DNS link under that domain.

Please contact support with details if you have any questions. – Andrea


12:46 am

Shared web server ‘tech21’ restored from backups on new hardware (RESOLVED)

Posted (December 25th, 2011 at 12:46 am PST) by Brian@dreamhost

We apologize for this outage, as the shared web server ‘tech21’ has been having hardware issues tonight. We decided to move it to new hardware and the restore process has been started.

The server should be running off of backups we’ve made, so sites should be up and working. If they’re not, please contact our support staff asap.

Please note, this is regarding the shared web server ‘tech21’ and no other services (email, MySQL) or machines are affected. To check if you’re on this server, please go to the DreamHost panel and check the home server for your user.

We will continue to update this post throughout the day! Please contact support if you are experiencing any issues with your account. Thanks for your continued patience! -Patrick

Update Dec 28th, 1:10pm PST: The restoration has completed, and all affected sites have now been restored from backups. Should you still be experiencing any issues, please contact our support team. -Brandon

Previous updates available after the jump →


1:11 am

MySQL VPS host ‘homie-vsql97′ maintenance scheduled for Jan 15, 2012 at 9PM PST (RESOLVED)

Posted (December 23rd, 2011 at 1:11 am PST) by chalise

MySQL VPS host machine ‘homie-vsql97′ will be offline for planned hardware maintenance on Sunday, January 15th, 2012 at 9:00PM PST. We estimate this will take about 30 minutes.

This affects the following MySQL VPS guests:
psmysql60371 psmysql60384 psmysql60403 psmysql60442 psmysql60443 psmysql60482 psmysql61318 psmysql61369 psmysql61739 psmysql61829 psmysql61841 psmysql61853 psmysql61857 psmysql61919 psmysql62571 psmysql62623 psmysql62650 psmysql62651 psmysql62660 psmysql62689 psmysql62710 psmysql62731 psmysql63090 psmysql72982 psmysql73024 psmysql73105 psmysql73107 psmysql73142 psmysql73216 psmysql73512 psmysql74167 psmysql74177 psmysql74222 psmysql74346 psmysql75007 psmysql75559

Please note, this is regarding the above MySQL VPS guests and no other services (such as email) or machines are affected (though it may cause brief downtime for any database driven websites relying on one of the above MySQL guests). To check if you’re on one of these MySQL guests, click “Account Status” in the top right hand corner of your DreamHost panel.

Update Jan 15th, 10:10pm PST: The maintenance on this server is complete and we’ve verified that things have been running smoothly for the last 15 minutes or so. If you’re having any further issues with a VPS server on this host, please contact support so that they can investigate it further for you. -dan w

Previous updates available after the jump →


8:03 pm

Web server ‘porto-novo’ offline intermittently due to denial of service attack (RESOLVED)

Posted (December 21st, 2011 at 8:03 pm PST) by Seohee

We are very sorry to report that the shared web server ‘porto-novo’ is once again offline intermittently due to a Denial of Service Attack.  Our Admins are working hard to get the issue handled and we hope to have it resolved shortly. We will provide more updates on the issue as soon as we can. Please contact Support with any questions.

Note: This is affecting just the ‘porto-novo’ web server and other services are still online (MySQL, e-mail, etc). No other servers are affected by this specific issue. To check if you’re on this server, please go to the DreamHost panel and check the home server for your user.

Update Dec 21st, 8:48pm PST: This issue is now resolved, thanks to the hard work of our Admins! Please contact Support if you notice any issues with your hosting on this server! -Seohee


8:09 pm

Web server ‘porto-novo’ offline intermittently due to denial of service attack (RESOLVED)

Posted (December 20th, 2011 at 8:09 pm PST) by Dan J

We are very sorry to report that the shared web server ‘porto-novo’ is offline intermittently due to a Denial of Service Attack.  Our Admins are working hard to get the issue handled and we hope to have it resolved shortly! We will provide more updates to this issue throughout the morning, or until the server is working again! Please contact Support with any questions!

Note: This is affecting just the ‘porto-novo’ server and other services are still online (MySQL, e-mail, etc). No other servers are affected by this specific issue.

Update Dec 21st, 12:12am PST: This issue is now resolved, thanks to the hard work of our Admins! Please contact Support if you notice any issues with your hosting on this server! -PatrickM


 
© 1996-2011, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
11:46 am

Web VPS host ‘homie-vserver147′ down due to hardware issues (RESOLVED)

Posted (December 20th, 2011 at 11:46 am PST) by Oscar

The raid card on the web VPS host machine ‘homie-vserver147′ has crashed and needs to be replaced. Our admin team is on route to replace the raid card. We will update this post as soon as that happens and hopefully we won’t need to run a file system check on the server. The affected machines are:

ps33734, ps38398, ps38433, ps38450, ps38456, ps38474, ps38530, ps38538,
ps38545, ps38587, ps38595, ps38637, ps38663, ps38882, ps39370, ps39374,
ps39504, ps39548, ps39702, ps39951, ps39953, ps40005, ps40440, ps42493,
ps43843, ps43859, ps44267, ps45455, ps47730, ps47829, ps48616, ps50225,
ps52505, ps53156, ps54133, ps54375, ps54507, ps55081, ps55257, ps55347,
ps55681, ps66386, ps71180, ps73434, ps74756

We will update this post as soon as our admin team is ready to replace the raid card.

Please note, this is regarding theweb VPS host machine ‘homie-vserver147′ and no other services (email, MySQL) or machines are affected. To check if you’re on this server, please go to the DreamHost panel and check the home server for your user.

Update Dec 20th, 1:58pm PST:This case is now resolved. The machine came back fine, and just to be sure, we rebooted it for a firmware upgrade. Everything appears to be in order again. If you continue seeing issues with your VPS, please contact support with the details, so we can assist you. – Andrea

Previous updates available after the jump →


6:26 am

WebFTP temporarily offline

Posted (December 20th, 2011 at 6:26 am PST) by chalise

Our WebFTP service is temporarily unavailable, and we have an admin is working on bringing it back up.

In the meantime, we do have a list of FTP clients to use on our FTP wiki page, here:

http://wiki.dreamhost.com/FTP#FTP_Client_Software

We will update this post again as updates become available. Thanks for your patience! – Chalise

Update 2011-12-20 8:09am PST: The issue that was causing webftp issues has been resolved, and all tests show it working normally again. If you are still seeing issues with webftp, please contact support.


12:34 pm

Shared web server ‘astana’ restoring from backups on new hardware (RESOLVED)

Posted (December 19th, 2011 at 12:34 pm PST) by andrea

We apologize for this outage, as the shared web server ‘astana’ has been having hardware issues this morning. We decided to move it to new hardware and the restore process has been started. The restore is using the last available backups, while the same backups are being used to run your sites.

The date of the backups being used is difficult to tell because of the restoring process itself, but if you’re having issues running into very old version of your site, please contact support with the details. We’ll be happy to assist to make sure you have the newest one we have available for you.

Please note, this is regarding the shared web server ‘astana’ and no other services (email, MySQL) or machines are affected. To check if you’re on this server, please go to the DreamHost panel and check the home server for your user.

Update Dec 23rd, 2:50pm PST: The restore is now 100% complete! We have restored all data from backups now and are resolving this post. Should you have any issues with your sites, please contact our support team and we will be glad to look into it for you. -Oscar Previous updates available after the jump →


7:52 pm

Shared web server ‘floyd’ being moved to new hardware (RESOLVED)

Posted (December 17th, 2011 at 7:52 pm PST) by mir

Due to a hardware failure we have had to find a new body for the shared server ‘floyd’ – our admin team has located a target and is in the process of failing floyd over to the new machine. We’ll update this post once it is back up and the data restore has begun. Thank you for your patience. -John

Update Dec 17th, 11:00pm PST: Our admins have moved the original disks over to new hardware and this has restored services on ‘floyd’. This issue has now been resolved but if you are still running into any problems please let support know. -JJ G Previous updates available after the jump →


2:18 pm

Shared web server ‘hagatna’ down for system file check (RESOLVED)

Posted (December 17th, 2011 at 2:18 pm PST) by Oscar

We apologize for the short notice, as the shared web server ‘hagatna’ has just been taken offline to complete a file-system integrity check. The server went read only after a raid card replacement.  This is affecting the root OS partition and not the customer data partition so the file system check should be done relatively quick. We will post more details about this as the check progresses!

Please note, this is regarding the shared web server ‘hagatna’ and no other services (email, MySQL) or machines are affected. To check if you’re on this server, please go to the DreamHost panel and check the home server for your user.

Update Dec 17th, 3:20pm PST: The fsck is complete and now your sites are back up and resolving. Should you have any issues with your sites being down, please contact our support team and will be glad to look into for you.-Oscar


11:07 am

Shared web server ‘johnson’ restoring from backups on new hardware (RESOLVED)

Posted (December 17th, 2011 at 11:07 am PST) by Oscar

We apologize for this outage, as the shared web server ‘johnson’ had to be taken offline to complete a file-system integrity check. The server was rebooted to resolve an issue with the file-system switching to read-only and this check needs to be completed to verify the integrity of the RAID array anytime this situation arises. Unfortunately, the server did not pass the initial check and we are now in the process of restoring this server’s data to new hardware while mounting user data from that backup server to keep sites online during the process.

The date of the backups being used is difficult to tell because of the restoring process itself, but if you’re having issues running into very old version of your data, please contact support with the details. We’ll be happy to assist to make sure you have the newest one we have available for you.

Please note, this is regarding the shared web server ‘johnson’ and no other services (email, MySQL) or machines are affected. To check if you’re on this server, please go to the DreamHost panel and check the home server for your user.

Update Dec 21st, 10:04pm PST: The restore is now complete! If you see that any of your data has not yet been restored, please do contact Support and let us know. Again, we sincerely appreciate your continued patience with this! -Seohee

Previous updates available after the jump →


5:17 am

Shared web server ‘warriors’ offline intermittently due to Denial of Service Attack (RESOLVED)

Posted (December 16th, 2011 at 5:17 am PST) by patrickm

We are very sorry to report that the shared web server ‘warriors’ is offline intermittently due to a Denial of Service Attack.  Our Admins are working hard to get the issue handled and we hope to have it resolved shortly! We will provide more updates to this issue throughout the morning, or until the server is working again! Please contact Support with any questions!

Note: This is affecting just the ‘warriors’ server and other services are still online (MySQL, e-mail, etc). No other servers are affected by this specific issue.

Update Dec 16th, 10:30am PST: This is now resolved as the server is up and running without any issues. Should you have any issues with your sites going down and you are on the shared web server ‘warriors’ please contact our support staff and we can look into it for you. – Oscar


2:20 pm

The minkar server is under DDoS attack (RESOLVED)

Posted (December 15th, 2011 at 2:20 pm PST) by danwy

The ‘minkar’ server is currently getting DDoS’d. Our admin team is aware of the problem and are working on mitigating the effects of it. We don’t have an ETA on when this will be resolved, but we will update this post as soon as we have any additional information regarding this situation.

Update 2011-12-15 2:45pm PST: The ip being targeted has been null-routed, and a new ip was given to the server. It is starting to recover now and we’ll continue to keep an eye on it. We are resolving this status post now, but if there are any further issues with sites on this server please don’t hesitate to contact support to let us know. –dan w


1:09 am

Shared web server ‘astana’ back online on old hardware (RESOLVED)

Posted (December 15th, 2011 at 1:09 am PST) by patrickm

We apologize for the short notice, as the shared web server ‘astana’ has just been taken offline to complete a file-system integrity check. The server was rebooted to resolve an issue with the file-system switching to read-only and this check needs to be completed to verify the integrity of the RAID array. We will post more details about this as the check progresses!

Please note, this is regarding the shared web server ‘astana’ and no other services (email, MySQL) or machines are affected. To check if you’re on this server, please go to the DreamHost panel and check the home server for your user.

Update Dec 15th, 3:20pm PST: We are now resolving this situation. The server has been running smoothly on the old hardware and all the affected sites we’ve checked are functioning again. If you continue to have problems with any domain on this server, please contact support to have them look into it further for you. -dan w

Previous updates available after the jump →


11:31 pm

Shared web server ‘sainthelier’ offline for unscheduled file-system check (RESOLVED)

Posted (December 14th, 2011 at 11:31 pm PST) by patrickm

We apologize for the short notice, as the shared web server ‘sainthelier’ has just been taken offline to complete a file-system integrity check. The server was rebooted to resolve an issue with the file-system switching to read-only and this check needs to be completed to verify the integrity of the RAID array. We will post more details about this as the check progresses!

Please note, this is regarding the shared web server ‘sainthelier’ and no other services (email, MySQL) or machines are affected. To check if you’re on this server, please go to the DreamHost panel and check the home server for your user.

Update Dec 15th, 4:42am PST: The file-system check has completed successfully! We’ve removed 2 of the drives from the array and that is being rebuilt on the back-end now, but services are all online for this machine! If you continue to have any issues, please contact Support! -PatrickM

Previous updates available after the jump →


 
© 1996-2011, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
6:57 pm

Network issues in Irvine datacenter causing delays to server updates (RESOLVED)

Posted (December 14th, 2011 at 6:57 pm PST) by Dan J

Our Irvine datacenter is currently having issues with our backend network. This is preventing some updates from the Dreamhost panel to servers in the Irvine datacenter. This includes things like FTP user creation, password changes, changes to website configurations done through the Dreamhost panel. The existing websites should still be online as the front end network is working properly. Our admin team is working as fast as they can to bring the backend network online again. All updates made in the Dreamhost panel should all go out once the backend network access has been restored. We apologize for the delays.

Update Dec 14th, 8:01pm PST: The Irvine backend network issues have been corrected now. Updates should be going to the servers in that datacenter now. Thank you for your patience. -Dan J

Previous updates available after the jump →


5:45 pm

Web VPS hosts homie-vserver265 and homie-vserver286 offline due to network issues (RESOLVED)

Posted (December 14th, 2011 at 5:45 pm PST) by Dan J

The web VPS hosts ‘homie-vserver265′ and ‘homie-vserver286′ are currently experiencing a networking issue that is preventing them from booting properly. Our administrators are working on the network to bring these servers back online. We will  make sure to update this post as soon as we can with further details.

Please note, this is regarding the web VPS hosts ‘homie-vserver265′ and ‘homie-vserver286′ and only the VPS on this host.  No other services (email, MySQL) are affected.

Update Dec 14th, 7:32pm PST: These two machines are now back online. If your site is not working properly then please contact tech support through the Dreamhost panel. -Dan J

Previous updates available after the jump →


4:21 pm

Web VPS host ‘homie-vserver24′ being restored from backups on new hardware (RESOLVED)

Posted (December 14th, 2011 at 4:21 pm PST) by Seohee

The web VPS host server ‘homie-vserver24′ was experiencing issues with the file-system switching to read-only mode, which is a sign of RAID issues. Our Admin team has decided to replace the hardware for this server and has begun a process of restoring data to this new hardware, while mounting the backups of all the VPS guests to keep sites online during the process.

This affects the following VPS guests:

ps10033 ps11042 ps11111 ps11156 ps11274 ps11279 ps11295 ps11583 ps11610 ps13071 ps13154 ps13931 ps14026 ps14117 ps14958 ps15014 ps15158 ps15161 ps35358 ps35360 ps37373 ps37797 ps38301 ps39245 ps53311 ps54076 ps55577 ps57678 ps6561 ps6615 ps6620 ps6810 ps6851 ps6915 ps69150 ps6939 ps6940 ps6971 ps6973 ps6977 ps7000 ps7010 ps7103 ps7119 ps7125 ps7179 ps72120 ps72236 ps7472 ps74838 ps7592 ps7615 ps8020

Please note, this is regarding the web VPS host ‘homie-vserver24′ and only the guest VPSs listed above.  No other services (email, MySQL) are affected.

Update Dec 18th, 3:30am PST: This restoration process has been completed and no further VPS machines are running off backups. This issue is resolved but if you have any trouble with a VPS on this machine please contact support. -JJ G Previous updates available after the jump →


2:59 am

Shared web server baham offline for RAID card replacement (RESOLVED)

Posted (December 14th, 2011 at 2:59 am PST) by chalise

The shared web server ‘baham’ was rebooted and is having trouble with its raid card, preventing it from coming back up. Our admins are in the process of replacing the raid card on this server.

Please note, this is regarding the shared web server ‘baham’ and no other services (email, MySQL) or machines are affected. To check if you’re on this server, please go to the DreamHost panel and check the home server for your user. – Chalise.

Update Dec 14th, 4:40am PST: baham’s raid card was successfully replaced and, after a firmware upgrade, the server has been successfully rebooted and is back up and running. If you have any further issues with sites on this server, please contact support so we can look into it for you! – Chalise.


2:10 am

Web VPS host ‘homie-vserver81′ offline for hardware issues (RESOLVED)

Posted (December 14th, 2011 at 2:10 am PST) by patrickm

The web VPS host server ‘homie-vserver81′ is currently offline due to RAID hardware issues. We are in the process of replacing this hardware now and will be providing another update shortly with more details on what is being done to resolve the problem.  This affects the following VPS guest servers:

ps13088 ps13323 ps13413 ps13425 ps13447 ps13458 ps13501 ps13510 ps13536 ps13554 ps13604 ps13634 ps13673 ps13751 ps13756 ps13877 ps13902 ps13919 ps13925 ps13936 ps13979 ps14031 ps14109 ps14209 ps14249 ps14313 ps14333 ps14393 ps14721 ps14788 ps14807 ps14941 ps14962 ps17963 ps18351 ps18352 ps22251 ps22319 ps28235 ps33592 ps33638 ps33954 ps34181 ps37112 ps43587 ps57512 ps61965

This will only cause issues for the web services for the above customers, so email and MySQL will remain active while the servers are offline.

Please contact Support with any questions and you can expect another update to this post very soon!

Update Dec 14th, 3:04am PST: The server has been moved to new hardware and its guests are back up and running. If you have any remaining issues with your sites, please contact support so we can look into it for you! – Chalise


3:35 pm

Shared web server ‘carina’ online and restoring to new hardware (RESOLVED)

Posted (December 13th, 2011 at 3:35 pm PST) by Dan J

The shared web server carina was taken offline for an unscheduled file-system check and at this time our Admin team decided to bring new hardware online and begin the process of restoring backups to that new server. To allow sites to continue to operate during this restoration process, backups of all domains have been mounted as live data, so that all sites should be online at this time.

The original hardware is still undergoing a file-system check, so that we can run a final data sync and provide everyone with the most current data possible. If you are having any issues with your sites on this server, please do not hesitate to contact Support from the “Support” section in your Web Panel.

Please note, this is regarding the shared web server ‘carina’ and no other services and machines are affected. To check if you’re on this server, please go to the DreamHost panel and check the home server for your user.

Update Dec 14th, 1:40pm PST: The admins have fixed the previous hardware hosting carina, all services have been restored. This was completed two hours ago, but we didn’t resolve the status post immediately to make sure everything was working correctly. Should you have any issues with your sites not resolving please contact our support staff and we will be glad to look into the issue for you. – Oscar Previous updates available after the jump →


5:25 pm

WordPress 3.3 now available in one-click installer

Posted (December 12th, 2011 at 5:25 pm PST) by Dan J

Customers who have used our one-click installer should be aware that WordPress 3.3 is now available. This is a feature release not a security release.

Dreamhost has made the decision to *not* run auto-upgrades until after the New Year. This is being done to prevent any possible complications with incompatible plugins on existing sites during the holiday season. Users can still upgrade via the Dreamhost panel or directly through the WordPress Dashboard.

Cool new features:

  • New Admin Bar
  • jQuery 1.7.1 Upgrade
  • Flyout menus
  • New Feature/Welcome highlights on install or upgrade
  • Drag and Drop multiuploader (now HTML5 compatible)
  • Improved Tablet Support in Admin

Full Release Notes can be found here:

http://wordpress.org/news/2011/12/sonny/

 

 

 


12:04 pm

Shared Web Server “bulls” offline due to hardware issues (RESOLVED)

Posted (December 12th, 2011 at 12:04 pm PST) by Brandon

The shared webserver “bulls” is currently offline due to file system issues on it’s operating system partition. We are currently re-imaging the partition, which can take several hours. We will keep this post up to date with information as it becomes available. -Brandon

Update Dec 12th, 4:30pm PST: The bulls server is back online. If you’re unable to reach your site then please contact support through the Dreamhost panel. Thanks. -Dan J

Previous updates available after the jump →


1:43 am

Nameservers receiving intermittent DOS attacks (RESOLVED)

Posted (December 11th, 2011 at 1:43 am PST) by Oscar

Our name servers ns1, ns2 and ns3.dreamhost.com are receiving intermittent denial of service attacks. Our admin team is working on the issue. This is the reason you may notice your sites not resolve for a brief period of time. We will update this post as soon as we have more information available.

Update Dec 12th, 9:15am PST: The nameservers have looked good since upgrading the hardware over two hours ago. This issue is now resolved, but if you’re experiencing issues with your sites please contact our Technical Support team.

Previous updates available after the jump →


7:07 am

VPS web host machine ‘homie-vserver119′ offline for file-system check (RESOLVED)

Posted (December 10th, 2011 at 7:07 am PST) by Oscar

The machine went into read-only mode. A filesystem will be required at this time. All services are unavailable at this time. Below is a list of the impacted VPS guests:

ps18642 ps18643 ps18644 ps18648 ps18661 ps18668 ps18681 ps18684 ps18692 ps18714 ps18723 ps18733 ps18739 ps18743 ps18970 ps19000 ps19141 ps19211 ps19230 ps19242 ps19244 ps19921 ps20988 ps21145 ps21244 ps21891 ps22021 ps22055 ps22561 ps24494 ps35369 ps36148 ps42601 ps43521 ps48823 ps52376 ps55751 ps55781

We sincerely apologize for any inconvenience this has caused and appreciate your continued patience. Please continue to check here for updates.

Update Dec 10th, 5:25pm PST: The old server is now back up and running and all guest machines are now up as well. At this point, if you have any issues with your sites being down, please contact our support staff and we will be glad to look into the issue for you. -Oscar Previous updates available after the jump →


 
© 1996-2011, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
8:58 pm

Shared web server ‘reykjavik’ restoring from backups to new hardware (RESOLVED)

Posted (December 8th, 2011 at 8:58 pm PST) by Seohee

The shared web server ‘reykjavik’ was offline after failing a file-system check. New hardware has been put in place and we are in the process of completing a live-restore on this server. This means that customer data is being mounted directly from the backup server and will act as live data for hosted sites on this server.

There may be periods of high load on the server, which is being closely monitored by our Admins. We will post frequent updates regarding the progress of the restore, as well as any issues that arise!

Please note, this is regarding the shared web server ‘reykjavik’ and no other services and machines are affected. To check if you’re on this server, please go to the DreamHost panel and check the “Machine” for your user.

Update Dec 21st, 3:47pm PST: We had to reopen this case, as many customers were missing some data. We re-run the restore, so please check again and let support know the details if you are still needing assistance. – Andrea

Previous updates available after the jump →


3:06 pm

Shared web server ‘reykjavik’ down for unscheduled file system check (RESOLVED)

Posted (December 8th, 2011 at 3:06 pm PST) by Dan J

The shared web server reykjavik is current down for an unscheduled file system check. We had some earlier issues where the file system went into read-only mode and in order to ensure the stability of the data on this machine the check must be executed.

Please note, this is regarding the shared web server ‘reykjavik’ and no other services and machines are affected. To check if you’re on this server, please go to the DreamHost panel and check the home server for your user.

We will post updates as this process progresses.

Update Dec 8th 3:15pm PST: The file system check on reykjavik completed without any errors and the server is now back online. If you’re unable to reach your site then please contact support. -Dan J


12:40 pm

Web VPS host ‘homie-vserver24′ offline for file-system check (RESOLVED)

Posted (December 7th, 2011 at 12:40 pm PST) by brians

The machine went into read-only mode. A filesystem will be required at this time. All services are unavailable at this time. Below is a list of the impacted VPS guests:

ps10033 ps11042 ps11111 ps11156 ps11274 ps11279 ps11295 ps11583 ps11610 ps13071 ps13154 ps13931 ps14026 ps14117 ps14958 ps15014 ps15158 ps15161 ps35358 ps35360 ps37373 ps37797 ps38301 ps39245 ps53311 ps54076 ps55577 ps57678 ps6561 ps6615 ps6620 ps6810 ps6851 ps6915 ps69150 ps6939 ps6940 ps6971 ps6973 ps6977 ps7000 ps7010 ps7103 ps7119 ps7125 ps7179 ps72120 ps72236 ps7472 ps7592 ps7615 ps8020

We want to do our best to eliminate any extended downtime. Our admins are currently prepping a temporary machine. Which will mount / serve data from our archival machine ( where your backups are stored ). This will allow your sites to load as the fschk is being performed on the old machine. The temporary machine should be online with a hour.

We sincerely apologize for any inconvenience this has caused and appreciate your continued patience. Please check for updates.

Update Dec 7th, 4:33pm PST: The issue with this server has been resolved! The file-system check completed successfully and the VPS are coming back online now. If your server or website is not responding then please contact support. -Dan J


3:13 am

Web VPS host ‘homie-vserver196′ being restored from backups on new hardware (RESOLVED)

Posted (December 7th, 2011 at 3:13 am PST) by patrickm

We apologize for this outage, as we had to take the web VPS host server ‘homie-vserver196′ offline for a RAID card replacement.  The new card was installed, but the server failed the file-system integrity check and we began the process of moving this server to new hardware. Unfortunately, the new hardware failed and we started over with another new server.  The restore process is now underway and, as usual, backups will be mounted during the restore to allow for services to remain online.

This affects the web services only for the following VPS guests:

prelude-relay-alc2 ps33593 ps33904 ps34490 ps34610 ps34657 ps34772 ps34792 ps34854 ps35101 ps35250 ps35333 ps35694 ps35797 ps36205 ps36290 ps36358 ps36449 ps36534 ps36721 ps36727 ps36786 ps36997 ps37225 ps37323 ps37334 ps37695 ps37709 ps37734 ps37789 ps38309 ps58609 ps58826 ps58894 ps60095 ps60169 ps60201 ps60212 ps60621 ps61239 ps62594 ps62598 ps62719 ps63005 ps63052 ps63177 ps63288

We will provide more updates as this process continues, but please contact Support if you have any questions in the meantime!

Update Dec 18th, 3:30am PST: This restoration process has been completed and no further VPS machines are running off backups. This issue is resolved but if you have any trouble with a VPS on this machine please contact support. -JJ G Previous updates available after the jump →


1:30 am

Shared Webserver Silversurfer Failing to Boot (RESOLVED)

Posted (December 7th, 2011 at 1:30 am PST) by jj

We’re currently having issues with the shared webserver silversurfer, our admins attempted to reboot the machine which is failing to come back up. We’ll post further update regarding this issue as soon as we have them.

Please note, this is regarding the shared web server silversurfer and no other services and machines are affected. To check if you’re on this server, please go to the DreamHost panel and check the home server for your user.

Update Dec 7th, 3:26am PST: The issue with this server has been resolved! The internal server which tells the web servers which kernel to use was providing incorrect data to the server when we rebooted it earlier. We fixed that up and the correct kernel is being used now and this server is back online! -PatrickM

Previous updates available after the jump →


1:28 am

Shared Webserver Tech21 Failing to Boot (RESOLVED)

Posted (December 7th, 2011 at 1:28 am PST) by jj

We’re currently having issues with the shared webserver tech21, at least one of the mounts reported as going read-only and when our admins attempted to reboot the machine fails to come back up. We’ll post further update regarding this issue as soon as we have them.

Please note, this is regarding the shared web server tech21 and no other services and machines are affected. To check if you’re on this server, please go to the DreamHost panel and check the home server for your user.

Update Dec 7th, 3:26am PST: The issue with this server has been resolved! The internal server which tells the web servers which kernel to use was providing incorrect data to the server when we rebooted it earlier. We fixed that up and the correct kernel is being used now and this server is back online! -PatrickM

Previous updates available after the jump →


11:21 am

Domain Registry Downtime (All TLD’s except .com/.net/.info/.org) (RESOLVED)

Posted (December 6th, 2011 at 11:21 am PST) by Justin@DH

We are currently in contact with our provider for the TLDs:

.co.uk .ca .us .biz .me .in .tv .eu .co.nz .de .org.uk .co .ws .mobi .name .cc .asia .me.uk .com.co .org.nz .net.nz .bz .mn .net.co .nom.co

We are seeing issues with registering new domains and transferring domains with these TLDs to us. Due to this, we are not able to process these registrations at this time. There is no website, email, mysql, dns or other downtime caused by this issue. When we have further information on when these will be available again, we will update this post. Our apologies for this inconvenience.

Update 2011-12-06 01:07pm PST: The issue has been found and resolved. Domain registrations can be attempted again. If you are still seeing issues, please contact support.

Note: Requested registrations from earlier are also being processed so there should be no need to re-request it. Please allow until tomorrow for us to get through them all, you will receive confirmation once we have processed the request for you (so if you have not heard from us by tomorrow it is safe to request the registration again and it should go through right away). Thank you for your patience!


4:07 am

Web VPS host server ‘homie-vserver195′ down for RAID replacement

Posted (December 6th, 2011 at 4:07 am PST) by chalise

Our admin team needed to replace the RAID card for this server, which began at 3:33am PST and completed at 3:51am PST. This affected the following VPS guests:

ps33598 ps33846 ps33938 ps34016 ps34202 ps34485 ps34517 ps35196 ps35238 ps35309 ps35314 ps35368 ps35417 ps35848 ps35996 ps36008 ps36079 ps36284 ps36365 ps36648 ps37243 ps37431 ps37435 ps37437 ps37757 ps37796 ps38228 ps45224 ps45829 ps46090 ps46708 ps46712 ps47280 ps47441 ps60154 ps60204 ps61276 ps64264 ps64752 ps65148 ps65586 ps65595 ps65622 ps65697 ps66082 ps71680 ps72697

Please note: This issue only affected the web services for the VPS guests listed above and will not affect email, MySQL or other services that do not depend on the web server.

The server is back up and running with no further issues; please contact support with any questions you may have.


8:32 am

Shared Webserver Porrima is suffering from a DDOS attack (RESOLVED)

Posted (December 5th, 2011 at 8:32 am PST) by Justin@DH

The shared webserver porrima is having issues due to an incoming DDOS attack. Our networking admins are working on the issue, to block, and stop the attack. When we have further information we will update this post, as soon as possible. This issue will cause website downtime. Email and mysql hosting will not be unaffected. This issue only affects the porrima webserver at this time.

Update 2011-12-05 9:03am PST: The ip being targeted has been null-routed, and a new ip given to the server. It is starting to recover now, but we are still making final fixes on it. We hope to have it back to normal running status shortly. – Justin

Update 2011-12-05 10:20am PST: The server has been running for 1.5 hours now, but still has a higher than normal load. We have our abuse team checking for any last issues. Sites should be running, except for those on the affected apache. They will need dns updates, which will take no longer than 4 hours for dns propagation. – Justin

Update 2011-12-05 10:50am PST: The server is now up 2 hours and seems stable, so we are resolving this post for now. If you are still seeing issues, please contact support.


1:40 am

Shared web server ’monrovia’ down for unscheduled file system check

Posted (December 4th, 2011 at 1:40 am PST) by Sandon

The shared web server monrovia is current down for an unscheduled file system check. We had some earlier issues where the file system went into read-only mode and in order to ensure the stability of the data on this machine the check must be executed.

Please note, this is regarding the shared web server ‘monrovia’ and no other services and machines are affected. To check if you’re on this server, please go to the DreamHost panel and check the home server for your user.

Update Dec 4 4:30AM PST: The file-system check has completed and all services are restored. Please contact support if you are still having problems with sites on ‘monrovia’ – Sandon


 
© 1996-2011, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
12:46 pm

Shared web server ‘redwings’ down for unscheduled maintenance (RESOLVED)

Posted (December 3rd, 2011 at 12:46 pm PST) by Oscar

The shared web server ‘redwings’ is currently down due to it’s root files system having an issue. We are currently running a file system check on it. As soon as it’s finished we will reboot the server and everything should be working again. This is not the customer data partition so the file system check should not take more than one hour.

Please note, this is regarding the shared web server ‘redwings’ and no other services or machines are affected. To check if you are on this server, please go to your Web Panel and check the home server for your user.

Update Dec 3 1:10pm PST: The file system check is finished and the server is back up now. Should you have any issues with your sites being down, please contact our support staff and we will be happy to look into it for you. -Oscar


5:10 am

Web VPS host server ‘homie-vserver196′ moved to new hardware and restoring from backups (RESOLVED)

Posted (December 2nd, 2011 at 5:10 am PST) by patrickm

The web VPS host server ‘homie-vserver196′ is had hardware issues and was taken offline for a file-system integrity check. When this was completed, we began the process of moving the server to new hardware. This affects the following VPS guests:

prelude-relay-alc2 ps33593 ps33904 ps34490 ps34610 ps34657 ps34772 ps34792 ps34854 ps35101 ps35250 ps35333 ps35694 ps35797 ps36205 ps36290 ps36358 ps36449 ps36534 ps36721 ps36727 ps36786 ps36997 ps37225 ps37323 ps37334 ps37695 ps37709 ps37734 ps37789 ps38309 ps58609 ps58826 ps58894 ps60095 ps60169 ps60201 ps60212 ps60621 ps61239 ps62594 ps62598 ps62719 ps63005 ps63052 ps63177 ps63288

Note: This issue only affects the web services for the VPS guests listed above.

The new server is live and has all data being mounted from backups while we complete a full data restore to the new hardware. All services should be online at this time, so please contact Support if that is not the case!

We will provide frequent updates as this process continues, so please check back for progress reports! Contact Support with any questions you may have!

Update Dec 2nd, 11:40pm PST: Great news! We were able to complete the original file-system check on the old hardware for this server and we have moved services back to that hardware! This means everything is back online for this VPS host and we do not need to wait for a lengthy restore to complete! Please contact Support with any questions!

Previous updates available after the jump →


1:56 am

Shared web server ‘zaurak’ moved to new hardware and restoring from backups (RESOLVED)

Posted (December 2nd, 2011 at 1:56 am PST) by patrickm

The shared web server ‘zaurak’ was taken offline for a file-system check this morning. This is being done to resolve issues with the RAID array in the server. Due to the frequency of issues with the disk array for this server, we have begun the process of swapping the hardware for the server. The new server is coming online now and will be mounting the backup server as live data for your sites, so that the server will remain online while we copy that backup data to the server itself.

Any uploads to the new server from this point on will not be overwritten by this process, so you may continue to make changes to your sites during this time.

Note: this outage is for the web server ‘zaurak’ only and will not affect email, MySQL or other services that do not depend on the web server.

Please check back for frequent updates on the progress of this process!

Update Dec 3, 7:40am PST: The restore is 100% complete. If you are missing any data, please contact our support team and we will be glad to check to see if we can restore that data for you. -Oscar Previous updates available after the jump →


5:27 pm

Shared server ‘yellowtail’ down due to hardware issue (RESOLVED)

Posted (December 1st, 2011 at 5:27 pm PST) by Seohee

The shared server ‘yellowtail’ is currently down due to a physical hardware issue. Our administrators have determined that it will need a chassis swap, and estimate that it will be up within an hour. We will make sure to update this post as soon as we have further details.

Please note, this is regarding the shared web server ‘yellowtail’ and no other services or machines are affected. To check if you are on this server, please go to your Web Panel and check the home server for your user.

We appreciate your patience!

Update Dec 1st, 7:40pm PST: The server is now back online. If you’re unable to reach your sites then please contact support. – Dan J

Previous updates available after the jump →


6:43 pm

Shared web server ‘jakarta’ will be offline at 9PM PST for scheduled maintenance (RESOLVED)

Posted (November 30th, 2011 at 6:43 pm PST) by Seohee

The shared web server ‘jakarta’  is currently experiencing an issue with the RAID, which manifests as the /home partition on the server becoming read-only. Details regarding this can be found in our previous post that was created earlier today.

http://www.dreamhoststatus.com/2011/11/30/shared-web-server-jakarta-having-hardware-issues/

We will start the failover to new hardware at 9PM PST tonight. We will make sure to continue to keep this post updated with its progress and further information.

Please note, this is regarding the shared web server ‘jakarta’ and no other services or machines are affected. To check if you are on this server, please go to your Web Panel and check the home server for your user.

We appreciate your patience!

Update Nov 30th, 10:18pm PST: Our admins report that this process has been completed and all services are running as expected. This post is now resolved but if you have any problems with your services on jakarta please let support know. –JJ G


5:58 pm

Shared server ‘matar’ down (RESOLVED)

Posted (November 30th, 2011 at 5:58 pm PST) by Seohee

The shared server ‘matar’ is currently down. Our administrators haven’t yet been able to pinpoint the cause of the outage, but we’ll make sure to update this post as soon as we have further details. We appreciate your patience.

Please note, this is regarding the shared web server ‘matar’ and no other services or machines are affected. To check if you are on this server, please go to your Web Panel and check the home server for your user.

Update Nov 30th, 10:30pm PST: Our admins have reported that the services to the server matar are now restored. There were some issues with the replacement of the original hardware which held up resolution, but that have now been fixed and sites are responding smoothly. If there are any problems with your services on matar please let support know. – JJ G


2:42 pm

Shared web server ‘jakarta’ having hardware issues (RESOLVED)

Posted (November 30th, 2011 at 2:42 pm PST) by patrickm

We are very sorry that we did not post this sooner, as we have been aware of hardware issues on the shared web server ‘jakarta’. Our Admin team identified an issue with the RAID for this server, which manifests as the /home partition on the server becoming read-only. The solution to this in the short term is a reboot of the server. However, when this issue was found, we began preparing new hardware for this server! That process has been underway for several days and we expect it to be completed soon!

We understand that these reboots are a hindrance and we are working to resolve this with a full hardware swap as soon as possible. We will post to this status blog again when the actual hardware change happens, with all relevant information.

Please note, this is regarding the shared web server ‘jakarta’ and no other services or machines are affected. To check if you are on this server, please go to your Web Panel and check the home server for your user.

Thank you for your continued patience! Please contact Support with any questions you may have!

Update Nov 30th, 10:22pm PST: Our admins report that the process of moving to the new hardware has been completed and all services are running as expected. This post is now resolved but if you have any problems with your services on jakarta please let support know. –JJ G Previous updates available after the jump →


8:06 am

.NZ Registry Downtime (scheduled maintenance, December 3, 2011)

Posted (November 30th, 2011 at 8:06 am PST) by mir

The .NZ Registry will be undergoing a planned downtime for maintenance on December 3, 2011, following this schedule:

Maintenance Details

Date: Saturday, December 3, 2011
Time: 06:00 PM – 08:00 PM (1800 hrs – 2000 hrs) GMT/UTC
Duration: 2 Hours

During the above maintenance window, you would not be able to Register, Renew, Transfer, Manage .NZ domain names with DreamHost.

This outage shall have no impact on the resolution/working of your already registered domain names. Please feel free to get in touch with our Support Team at through the panel (https://panel.dreamhost.com/index.cgi?tree=support.msg&) or directly (support@dreamhost.com), in case of any queries/doubts. -John


8:06 pm

Shared web server ‘alembic’ offline for unscheduled RAID card replacement (Resolved)

Posted (November 29th, 2011 at 8:06 pm PST) by Seohee

The shared web server ‘alembic’ has a bad raid card and is not functioning properly as a result. Our admins are working on replacing the card on this machine. We will update this post as soon as we have more details and apologize for the inconvenience.

Please note, this is regarding the shared web server ‘alembic’ and no other services and machines are affected. To check if you’re on this server, please go to the DreamHost panel and check the home server for your user. -Seohee

Update Nov 29th 8:17PM PST: The shared server ‘alembic’ is now running on a new raid card. The server and all sites are now functional. Please feel free to contact Support should you notice any issues with your sites. We appreciate your patience while we got this worked out! -Seohee


3:01 pm

Networking issues affecting a row of DEDICATED machines in our LAX datacenter ( RESOLVED )

Posted (November 29th, 2011 at 3:01 pm PST) by brians

We are currently experiencing networking issues with a particular rack of DEDICATED machines at our LAX datacenter. Our admin team is on site and currently working on the issue. The affected machines are below:

 argtracker, armato, bandr, billbiz, bmw, dixie, ds1172, ds1212, embassy, evangeline, factorycat, fepi, gazoline, greyson, idpfilm, jupiter, kmi,
 lowest-airfare, motmot, mqseries, nwtekno, perfecto, phobos, prometheus, reallifelab, shodan, sitepartner, timber

The switch was rebooted and the servers are back online. Our network engineers will continue to keep any eye on things. The post will be resolved once we are confident that the issue is indeed resolved. We sincerely apologize for any inconvenience this has caused. Please check back for updates.

UPDATE Nov 29th 3:50PM PST: The switch reboot took care of the network problem. All services on your dedicated machines are fully functional. Sorry for any inconvenience the outage may have caused. If you are having problems, please contact Tech Support. – Brian S


 
© 1996-2011, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
8:05 am

Sending of Mail Messages from Sites and Scripts Delayed (RESOLVED)

Posted (November 29th, 2011 at 8:05 am PST) by Oscar

Due to a spam run on a few of our mail servers, a small number of our outgoing mail servers have extremely large queues which are resulting in delays with mail be sent.  This will not prevent mail from being sent out however, it will cause significant delays in web based email being sent. This includes emails sent from your sites as well as scripts or crons. This is not affecting mail server via SMTP. Our admin team is currently working on the issue and we will update this post as soon as we have more information.

Update Nov 30th, 10:15am PST: Our abuse team has done a lot of work to clean up the mail queues on the webserver affected, and mail delivery should be back to normal now. If you are still seeing issues, please contact support. – Justin

Previous updates available after the jump →


1:42 pm

Shared webserver “hydra” restoring from backups (RESOLVED)

Posted (November 27th, 2011 at 1:42 pm PST) by Justin@DH

The shared web server ‘hydra’ is having unrecoverable hardware issues that is requiring us to setup a new webserver and restore to it from backups. The webserver will run off the backups while it restores, so sites will be working and usable.

Please note, this is regarding the shared web server ‘hydra’ and no other services and machines are affected. To check if you’re on this server, please go to the DreamHost panel and check the home server for your user.

Update Dec 3 3:05pm PST: This restore is now complete. If you are missing any data, please contact our support staff and we will be glad to check our backups for you. -Oscar Previous updates available after the jump →


4:09 pm

Shared web server ‘noumea’ offline for unscheduled RAID card replacement (RESOLVED)

Posted (November 26th, 2011 at 4:09 pm PST) by mir

The shared web server ‘noumea’ has a bad raid card and is not functioning properly as a result. Our admins are are working on replacing the card on this machine. We will update this post as soon as we have more details and apologize for the inconvenience.

Please note, this is regarding the shared web server ‘noumea’ and no other services and machines are affected. To check if you’re on this server, please go to the DreamHost panel and check the home server for your user. -John

Update Nov 26th 4:40pm PST: Our admin team is still working on this issue, we’ll update this post once we have more information for you. -John

Update Nov 26th 5:14PM: Nourmea is now running on a new raid card. All sites are now functional. Feel free to contact support should you notice any issues with your site.  -Phiya


8:19 am

.ORG Domain Registry Maintenance (RESOLVED)

Posted (November 26th, 2011 at 8:19 am PST) by jj

On November 26, 2011 from 08:00am to 04:00pm PST the .org domain registry will be performing maintenance. During this time frame registrations, renewals, updates and availability checks, will not be possible. DNS and whois services will continue to work properly.

Thank you for your patience while the maintenance is underway!

Update Nov 26th 2:02pm PST: This should now be fixed! If you’re still seeing problems making changes to registrations, please contact support. — Brian H


6:57 am

Shared web server ‘zaurak’ down for unscheduled file system check (RESOLVED)

Posted (November 26th, 2011 at 6:57 am PST) by jj

The shared web server zaurak is current down for an unscheduled file system check. We had some earlier issues where the file system went into read-only mode and in order to ensure the stability of the data on this machine the check must be executed.

Please note, this is regarding the shared web server ‘zaurak’ and no other services and machines are affected. To check if you’re on this server, please go to the DreamHost panel and check the home server for your user.

We will post updates as this process progresses.

Update Nov 26th 7:42am PST: The services on zaurak have been up and running smoothly for over 15 minutes and show no signs of problems. This issue has now been marked as resolved but if you have any problems with a site on this server please let support know. -JJ G Previous updates available after the jump →


2:29 am

Shared web server ‘dorado’ offline for RAID card replacement (RESOLVED)

Posted (November 26th, 2011 at 2:29 am PST) by jj

The shared web server ‘dorado’ was rebooted earlier and is having trouble with the raid card, preventing it from coming back up. Our admins are going to replace the card on this machine.

Please note, this is regarding the shared web server ‘dorado’ and no other services and machines are affected. To check if you’re on this server, please go to the DreamHost panel and check the home server for your user.

Update Nov 26th 3:00am PST: This server looks stable and happy, so this issue is being marked as resolved. If you have any problems with this machine please let support know. -JJ G Previous updates available after the jump →


9:52 pm

Shared server ‘equuleus’ offline for unscheduled RAID card replacement (RESOLVED)

Posted (November 25th, 2011 at 9:52 pm PST) by Dan J

Our admins report that the shared web server equuleus has gone offline due to it rejecting I/O and having boot failures, the RAID card is now being replaced. We will update this post with further information once the replacement has been put in place. -Dan J

Update Nov 25th, 10:03pm PST: The equuleus server is back online. The RAID card swap was successful with no complications. If your site is not back online then please contact support. -Dan J


9:00 pm

Web VPS host ‘homie-vserver213′ unscheduled file system check (RESOLVED)

Posted (November 25th, 2011 at 9:00 pm PST) by Dan J

The VPS host server ‘homie-vserver213′ is in the process of being moved over to new hardware. It went into read-only mode and is now running a file system check. Our admins are setting up a second machine using backups to get the servers back online while the file system check completes. This affects the following VPS guest servers:

ps44679 ps44766 ps44859 ps44887 ps44892 ps44901 ps44903 ps44904 ps44908 ps44911 ps44912 ps44926 ps44933 ps44936 ps44939 ps45101 ps45197 ps45380 ps45381 ps45626 ps45678 ps45683 ps45783 ps45833 ps45903 ps45993 ps45997 ps46111 ps46373 ps46418 ps46447 ps46730 ps46826 ps47232 ps47236 ps47245 ps47564 ps47565 ps47769 ps47783 ps47798 ps48236 ps48297 ps48449 ps48756 ps49431 ps49840 ps49859 ps49955 ps50292 ps53660 ps53966 ps54476 ps55255 ps55714 ps56534 ps56565 ps65090 ps65209 ps71566

Please note, this issue is only affecting the web data on the machines listed above, and no other services and machines are affected. To check if you’re on this server, please go to the DreamHost panel and check the home server for your user (or just check your VPS name).

Please contact Support from the “Support > Contact Support” page in your Web Panel if you have any questions!

Update Nov 26th 12:20am PST: This file system check is now completed and services have been up and running smoothly. This issue is resolved but if you run into any problems with your VPS on this machine please let support know. -JJ G Previous updates available after the jump →


8:29 pm

Shared server ‘mbabane’ offline for unscheduled RAID card replacement (RESOLVED)

Posted (November 25th, 2011 at 8:29 pm PST) by Dan J

Our admins report that the shared web server mbabane has gone offline due to it rejecting I/O and having boot failures, the RAID card is now being replaced. We will update this post with further information once the replacement has been put in place. -Dan J

Update Nov 25th, 9:30pm PST: The mbabane server is back online. The RAID card swap was successful with no complications. If your site is not back online then please contact support. -Dan J


6:17 pm

Shared web server ‘yankees’ restoring from backups (RESOLVED)

Posted (November 25th, 2011 at 6:17 pm PST) by Oscar

The shared web server ‘yankees’ is having issues rebooting and it looks to be an issue with the raid array on the server. If the raid array cannot be recovered, we will need to fail over this machine. Our admin team is working on the server at this time and we will update this post as soon as we have more information.

Please note, this is regarding the shared web server ‘yankees’ and no other services and machines are affected. To check if you’re on this server, please go to the DreamHost panel and check the home server for your user.

Update Dec 2nd, 5:56pm PST: Good news! The restore on the yankees server has completed. The issue with paths and symlinks have been cleaned up and the server should be fully operational. If you’re unable to view your sites or have any other issues then please contact support. -Dan J

Previous updates available after the jump →


 
© 1996-2011, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
3:55 pm

Irvine Datacenter Networking Issue Affecting Specific Webservers (RESOLVED)

Posted (November 23rd, 2011 at 3:55 pm PST) by Dan J

We are seeing issues with a row of servers in the Irvine datacenter. We believe the issue to be networking related downtime. We have our admins on the issue, trying to find the cause and fix it. Our apologies for the issues with these specific services. We will continue to update here with more info as it comes in. – Dan J

Update Nov 23rd, 4:20pm PST: The issue have been resolved. Our admins reported that a new component was plugged in before it’s ports were configured. The new component had the same configuration as an existing row of servers and that caused the ports for that row to shut down. We apologize for the downtime. Everything should be back online now. If you’re still seeing any issues with your server then please contact the support team. – Dan J


2:34 pm

Problems with some features on the main DreamHost.com website (RESOLVED)

Posted (November 22nd, 2011 at 2:34 pm PST) by andrea

We’ve been having some problems with some parts of DreamHost.com since this morning. Our system administration team is aware and is working towards the resolution. This is only affecting that main site, and all DreamHost services are operational, including the DreamHost panel. Just wanted to give our customers the heads up. Sorry about any inconvenience this may be causing. We will update this post with any progress we make. – Andrea

Update 2011-11-22 4:41PM PST: Our admin team are reporting that the DreamHost.com website should be working fine now. The issues appear to have been caused by some new updates and have been corrected. – Dan J


12:07 pm

Web VPS host homie-vserver218 moving to new hardware on Nov 22nd at 9pm PST (RESOLVED)

Posted (November 22nd, 2011 at 12:07 pm PST) by danwy

The vserver host homie-vserver218 has been having repeated issues with going into a ‘read only’ state, requiring multiple reboots to fix. As a result, we have prepared new hardware to move this server to and have copied over data to it already. At 9pm (PST) tonight, we will be switching the server over to the new hardware in an effort to improve stability on it. There should be about an hour of downtime and we will do a final rsync when that is complete to copy over any new files that have been uploaded since our initial rsync finished about an hour ago. Here are a list of servers that will be affected:

ps20165 ps45532 ps45536 ps45540 ps45553 ps45586 ps45593 ps45634 ps45664 ps45676 ps45780 ps45782 ps45827 ps45975 ps46050 ps46054 ps46069 ps46141 ps46181 ps46235 ps46453 ps46575 ps46823 ps46892 ps47265 ps47492 ps47590 ps47649 ps47824 ps49690 ps49919 ps50041 ps50068 ps54035 ps65035 ps65174 ps65407 ps65475 ps65699 ps65701 ps66085 ps68717 ps70459

If your server is not listed above and goes down at 9pm (PST), please contact support to investigate it further for you as it is not related to this maintenance.

Update Nov 22nd, 10:23pm PST: This hardware change is complete and all the above VPS guests should now be back online! If your VPS is not online, or not behaving correctly, please contact Support for assistance! -PatrickM

Previous updates available after the jump →


9:55 am

Downtown LA Datacenter Networking Issues For Specific Webservers (RESOLVED)

Posted (November 20th, 2011 at 9:55 am PST) by Justin@DH

We are investigating a networking issue that is causing outages for a specific row of servers in our downtown LA datacenter. Possible servers that we believe could be affected at this point are listed below. We have our admin team on the case to figure out the cause and fix it as soon as we can. We will update this post when we have further info.

amber angel argos arlene aslan avarice bagheera bang barbaro barfolomew bennett bigglesworth bill bishop blade buck burbank cannonball cassandra catbert cerberus cinnamon clarinet clifford concertina constantine courage cykotix domination dredd duchess ein fang fluffy fwap garm ghostrider greenlantern grendel growltiger harmonica hawkman hazelnut hercules hobgoblin homie-vserver177 homie-vserver231 homie-vserver25 homie-vserver36 homie-vserver38 homie-vserver39 homie-vserver40 homie-vserver41 homie-vserver49 homie-vserver50 homie-vserver51 homie-vserver52 homie-vserver54 homie-vsql02 homie-vsql03 homie-vsql04 homie-vsql05 homie-vsql06 homie-vsql07 homie-vsql08 homie-vsql09 homie-vsql25 homie-vsql26 homie-vsql27 homie-vsql28 homie-vsql29 homie-vsql40 homie-vztest humantorch hushpad hushpuppy jake joecool jones koi kona krypto lamprey lassie leo lingling lionel luthor masevo masq midnight moran mrfantastic muffy mystique namor naroga neko odie petrelli preacher proty puff putney ren roary sabretooth sawdust scooby seabiscuit sekhmet shadow silversurfer sirius sizzle skip snagglepuss snapper snarf snowdrop sparky spawn spike sprocket sui tick tiny tomoe tramp trombone trumpet twoface ubu underdog vuvuzela wasp watanabe wonderwoman yeller

Update 2011-11-20 10:00AM PST: We have heard word from the network admins that the outage is over, but we are still verifying that everything is back up and working normally. We will be investigating the cause and will provide more info shortly. – Justin Previous updates available after the jump →


8:00 pm

dreamhost.com website down (RESOLVED)

Posted (November 19th, 2011 at 8:00 pm PST) by Oscar

Our admin team is working on the site as it’s currently down. This is no affecting any web servers or  the customer web panel. Again, this is only affecting our sales website dreamhost.com. If you need to log into your web panel you can do so via the panel link here:

 

We will update this post as soon as we have more information.

UPDATE Nov 19th 07:50pm PST: This should now be taken care of as of Nov 18th 6pm PST– Brian H


11:04 am

Shared server ‘lion-o’ restoring from backups (RESOLVED)

Posted (November 19th, 2011 at 11:04 am PST) by Oscar

The shared server ‘lion-o’ went in read-only mode. Upon rebooting, the server would not come back up so we are running a file system check (fsck) on the server. We will be mounting backups to a new machine while the old machine completes the fsck. We will update this post as soon as we have the new machine up and backups mounted.

Please note, this is regarding the shared web server lion-o, and no other services and machines are affected. To check if you’re on this server, please go to the DreamHost panel and check the home server for your user.

Update Nov 20th 8:20pm PST: We’re happy to report that the restore process is now complete, no users are running off backups any more.

If you’re still having issues with your site on lion-o, it’s unrelated, so please contact support with the details, so we can assist you. – Andrea

Previous updates available after the jump →


9:30 am

Shared server ” bunda ” being restored from backups (RESOLVED)

Posted (November 18th, 2011 at 9:30 am PST) by brians

The shared web hosting server ‘bunda’ has been taken offline for an unscheduled file-system check. The server rebooted to a read-only file-system, which is a sign of issues with the drives, so this check is being performed to verify the integrity of the data on the server. We will update this post as this process continues and once we have an ETA.

UPDATE Nov 20th 10:30am PST: The restore has completed. If you are still seeing issues after this restore, please contact support. – Justin

Previous updates available after the jump →


12:43 am

Shared web server ‘kalamazoo’ being moved to new hardware (RESOLVED)

Posted (November 18th, 2011 at 12:43 am PST) by jj

The shared server ‘kalamazoo’ is currently having issues with the file system and our admins are moving it over to new hardware. Services will be brought up and restored from backups made on 11-17-2011. We will continue to post updates regarding this issue as they become available.

Please note, this is regarding the shared web server kalamazoo, and no other services and machines are affected. To check if you’re on this server, please go to the DreamHost panel and check the home server for your user.

Update Nov 21st, 1:45am PST: This server is completely restored from backups! Please contact Support if you notice any issues with your sites hosted on this server! Thanks for your patience! -PatrickM

Previous updates available after the jump →


9:45 pm

Shared server ‘apocalypse’ offline for unscheduled RAID card replacement (RESOLVED)

Posted (November 17th, 2011 at 9:45 pm PST) by jj

Our admins report that the shared web server apocalypse has gone offline due to it rejecting I/O and having boot failures, the RAID card is now being replaced. We will update this post with further information once the replacement has been put in place.

Update Nov 17th, 11:00pm PST: This issue is now resolved, everything has been running well and the server seems to be stable. If you continue to have any problems with a website on this machine please let support know.  -JJ G Previous updates available after the jump →


2:46 pm

Web VPS host ‘homie-vserver178′ being restored from backups on new hardware (RESOLVED)

Posted (November 17th, 2011 at 2:46 pm PST) by danwy

The VPS host server ‘homie-vserver178′ is in the process of being moved over to new hardware. It went into read-only mode and we were not able to fix it with either a filesystem check or by replacing its RAID card. This affects the following VPS guest servers:

ps16220 ps16223 ps16264 ps16306 ps16315 ps16831 ps17437 ps18736 ps27905 ps28045 ps28066 ps28071 ps28143 ps28155 ps28160 ps28239 ps28364 ps28397 ps28408 ps28461 ps28462 ps28636 ps28793 ps29142 ps30698 ps33699 ps33777 ps34157 ps34433 ps34445 ps34559 ps35078 ps35118 ps35662 ps37223 ps39619 ps41978 ps42660 ps43577 ps44741 ps45134 ps46743 ps63944 ps69338

Please note, this issue is only affecting the web data on the machines listed above, and no other services and machines are affected. To check if you’re on this server, please go to the DreamHost panel and check the home server for your user (or just check your VPS name).

Please contact Support from the “Support > Contact Support” page in your Web Panel if you have any questions!

Update Nov 28th, 10:30am PST:The restore has completed and has been confirmed as completed by one of our admins. If you see further issues with your VPS on this host machine, please contact support. – Justin

Previous updates available after the jump →


 
© 1996-2011, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
10:25 am

Oboe mysql server receiving a DDOS attack (RESOLVED)

Posted (November 17th, 2011 at 10:25 am PST) by Justin@DH

The oboe mysql server is currently having issues due to networking and a DDOS attack. We have given the service with the ip being attacked a new ip, and are working to block the other ip from our network. This has caused some downtime for customers on the oboe mysql server recently, and we are still working to make sure it is cleaned up. We will update this post when we have further info on the status of the cleanup.

Update 2011-11-17 11:00am PST:The ip being attacked has been blocked and a new ip given to oboe:zahl, the server is stable again and we are keeping an eye on it. If you notice further issues, they could be unrelated, in that case please contact support with specific details.


7:48 am

Irvine Datacenter Networking Issue Affecting Specific Webservers (RESOLVED)

Posted (November 17th, 2011 at 7:48 am PST) by Justin@DH

We are seeing issues with the following webservers in the irvine datacenter:

miram mirfak mizar mothallah muliphein muphrid muscida nair-al-saif naos nashira navi nekkar nembus neshmet nihal nunki nusakan okul peacock phact pollux porrima praecipua procyon propus proxima-centauri pulcherrima rana ras-algethi ras-alhague rasalas rastaban sulafat syrma tabit talitha-australis talitha-borealis tarazet taygeta

We believe the issue to be networking related, as they are having slowness connecting, issues connecting to their mysql server, and other downtime. We have our admins on the issue, trying to find the cause and fix it. Our apologies for the issues this morning with these specific services.

Update 2011-11-17 08:50am PST:After investigation, we have found a site that was a target of a large DDOS attack and we have blocked and taken appropriate action for this domain name, to prevent it causing further issues on our hosting. We are seeing networking returning back to normal and believe the issue now resolved. If you have further issues please contact support.


3:22 pm

VPS host ‘homie-vserver168′ offline for unscheduled hardware swap (RESOLVED)

Posted (November 16th, 2011 at 3:22 pm PST) by Dan J

The VPS host machine homie-vserver168, has stopped responding.  Our admins have determined it needs it’s raid card replaced. One of our team is heading down to replace that now. The affected VPS servers are:

ps23373 ps23397 ps23496 ps23526 ps23563 ps23640 ps23644 ps23677 ps23715 ps23787 ps23918 ps23960 ps24007 ps24303 ps24339 ps24419 ps24480 ps24481 ps24484 ps24697 ps24703 ps24711 ps24719 ps24752 ps24889 ps24911 ps25479 ps25644 ps28450 ps29119 ps29277 ps29289 ps29375 ps29736 ps34452 ps34465 ps34645 ps37251 ps38207 ps38339 ps38525 ps39096 ps39233 ps39427 ps40054 ps46545 ps46584 ps46589 ps46662 vserver24501

Update Nov 16th, 5:30pm PST: The hardware swap went well and the server is now back online. If you are still having any trouble then please contact support.


11:55 am

Email Outage on Sub1 cluster of mail service due to MySQL problem (RESOLVED)

Posted (November 16th, 2011 at 11:55 am PST) by Ralph

Mail is down for our sub1 cluster of email. You can find if you are affected by checking the “Account Status” link near the top right of the page when you are logged into the DreamHost Web Panel. Homiemail-sub1 will be listed as “Your Email Server”.

Our MySQL database that handles the back end authorization for logging in and delivering mail is currently down and being worked on by our administration team. Our ETA for resolution is 2 hours. Your mail will not be lost, it will simply be delayed. You will also not be able to log in during this outage. We hope to have the issue resolved in less than 2 hours.

Update Nov 16th, 11:58am PST: Issue is now considered resolved. We will keep an eye on the service to make sure no further problems continue. We had a mail issue this morning as well that cleared up before we were able to fix it. We believe both to be related though this time we noticed the space issue with the MySQL server and we have fixed the problem. We do not expect any further problems to continue. Apologies for the temporary drops in service this morning.


11:14 am

Shared server ‘tashkent’ offline for unscheduled RAID card replacement (RESOLVED)

Posted (November 16th, 2011 at 11:14 am PST) by mir

We are sorry for the short notice but the RAID controller in the ‘tashkent’ web server has gone bad and requires replacement. The downtime from this should be less than 1 hour. We will update this post when this is complete, or if there are any relevant updates during the process.

Update Nov 16th, 12:30pm PST:Tashkent is back up and running. We tested some sites on the server to make sure everything is running as expected. Please accept our apologies for any inconvenience, and let support know if you need further assistance, as it’s likely unrelated. – Andrea

Previous updates available after the jump →


11:15 pm

VPS host ‘homie-vserver24′ live restore from backups due to RAID issues (RESOLVED)

Posted (November 15th, 2011 at 11:15 pm PST) by patrickm

The VPS host server ‘homie-vserver24′ has been moved to new hardware due to a failure with the RAID array on this machine. We are performing a live restore of the server, which means that all services will remain active while data is copied from the backup server for each of the following VPS guest servers:

ps10033 ps11042 ps11111 ps11156 ps11274 ps11279 ps11295 ps11583 ps11610 ps13071 ps13154 ps13931 ps14026 ps14117 ps14958 ps15014 ps15158 ps15161 ps35358 ps35360 ps37373 ps37797 ps38301 ps39245 ps53311 ps54076 ps55577 ps57678 ps6561 ps6615 ps6620 ps6810 ps6851 ps6915 ps6939 ps6940 ps6971 ps6973 ps6977 ps7000 ps7010 ps7103 ps7119 ps7125 ps7179 ps7472 ps7592 ps7615 ps8020

We will update this post with details as this process continues! Please contact Support from the “Support > Contact Support” page in your Web Panel if you have any questions!

Update Nov 20th, 7:00am PST: The restore has completed. If you are seeing any issues since the restore completed, please contact support. – Justin

Previous updates available after the jump →


8:23 am

Shared webserver “brewers” hardware issues (RESOLVED)

Posted (November 15th, 2011 at 8:23 am PST) by Justin@DH

The shared webserver brewers is having issues with its root file system, which contains the operating system. Our admins are working to recover the OS partition at this time. Should this not work out, we will have to re-image (re-install) the OS. Currently the data partition that contains customer data is not affected, but should this situation change we will update this post.

Update Nov 15th, 8:35am PST: The file system check on the OS partition has completed and the server booted up fine. We believe the issue is resolved, if you still see any issues please contact support.


11:41 pm

Possible network issue affecting MySQL connections (RESOLVED)

Posted (November 14th, 2011 at 11:41 pm PST) by patrickm

We are currently experiencing an issue which is affecting MySQL connections for many customers.  Our Network and Admin teams are investigating this issue and we hope to have it resolved shortly.  I will update this post as soon as we have any additional information.  Feel free to contact Support if you believe you are affected by this outage!

Update Nov 15th, 12:35am PST: This issue is now completely resolved! Scheduled maintenance with one of our providers was not supposed to cause any issues for customer-facing connections, but that did not go as planned. The maintenance was reversed and canceled for tonight, so that more investigation can be completed as to why this caused this outage. We have tested and all connections are now working again, so please contact Support if you continue to have any issues! -PatrickM


9:43 pm

Shared server ‘dili’ on new hardware and restoring from backups (RESOLVED)

Posted (November 14th, 2011 at 9:43 pm PST) by Seohee

The shared server ‘dili’ was offline due to bad data drives. Drive 1 was experiencing issues and drive 2 failed, and because they are on the same set, they could only be swapped one at a time. Our admins hoped that drive 1 would hang in there until drive 2 was done rebuilding, but luck wasn’t on our side.

New hardware is in place and backups are being mounted as live data while we restore all data to the new server. All user data is mounted from the backup server using the latest backup before this issue began (which should be Nov 14th). Any data uploaded at this point will be merged with the backup data at the end of this process.

This post will be updated throughout this process. Please check back for frequent updates!

Please note, this post is about the shared web server ‘dili’, and doesn’t affect anything else but the web file restoration on that machine. To find out if you’re on dili, go to the DreamHost Panel and check the home for your FTP/shell user.

Our apologies for the inconvenience this has caused. Please feel free to contact Support if you have any questions.

Update Nov 19th, 3:00am PST: We ran into a problem with the restoration process and the restore had stopped running for awhile. Luckily there was very little left to do and the process has now been completed. This issue has been resolved but if you are having any issues with a site on Dili please let support know. -JJ G Previous updates available after the jump →


12:09 pm

Networking issues in one of our datacenters (RESOLVED)

Posted (November 14th, 2011 at 12:09 pm PST) by danwy

We’re currently looking into some networking issues in our Irvine datacenter. If your site is down now, that is likely the cause. Our network admin is already investigating the issue and hope to have it back ASAP. We will update this status post shortly with a list of affected servers and an ETA on when we expect it to be resolved.

Update Nov 14th 1:40pm PST:The cause of the outage was determined to be an issue with the failover mechanism – basically one controller overreacting and thus causing networking confusion. Our admins are investigating how this happened and will make sure it won’t happen again. We’re currently answering your support messages about this, so you will hear back from us shortly.
If you’re still having issues with your sites, it’s unrelated, so please contact support with the specifics, so we can assist you. – Andrea

Previous updates available after the jump →


 
© 1996-2011, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
3:35 am

Shared server ‘jaga’ going offline for unscheduled maintenance (RESOLVED)

Posted (November 14th, 2011 at 3:35 am PST) by patrickm

We are very sorry to report that the RAID card for the shared server ‘jaga’ is acting up and we need to take the server offline to replace that hardware. We do not expect this will take very long and we will update this post as soon as the maintenance has been completed! Please contact Support with any questions you may have!

Update Nov 14th, 5:44am PST: Sorry for the delay in our resolution to this post! The process took a bit longer than expected, but the server has been back online for almost an hour with the new hardware! Please contact Support if you notice any issues with your hosting on this server! -PatrickM


10:36 am

Webserver VPS Creation Delays (RESOLVED)

Posted (November 13th, 2011 at 10:36 am PST) by Justin@DH

We are seeing a backlog of requested new VPS not being created in a timely manner. We have developers and admins looking into this issue currently, to fix the issue that is preventing the VPS from being created and assigned to accounts that requested them. If you recently (within the last couple days) requested a new webserver VPS and it has not yet completed or emailed you that it is added, you may be affected. We will update this post when the creation is working properly again.

Update Nov 14th 10:15am PST: I’m happy to report that the VPS that were requested yesterday have been all created. The currently pending ones are from this am, and they are being processed as usual. We’re also working on speeding up this process for the future. We appreciate your patience. – Andrea

Previous updates available after the jump →


10:33 am

Vserver host ‘homie-vserver219′ offline for unscheduled hardware swap (RESOLVED)

Posted (November 13th, 2011 at 10:33 am PST) by Justin@DH

The VPS host machine homie-vserver219, has stopped responding. An attempt to remotely reboot the server has not worked, so it is being manually rebooted first. If this does not fix the issue, it may need a raid card replaced. We will update this post when we have more information. The affected VPS servers are:

ps20024 ps22705 ps38861 ps39743 ps45526 ps45530 ps45535 ps45649 ps45741 ps45755 ps45803 ps45872 ps46036 ps46051 ps46160 ps46195 ps46215 ps46216 ps46222 ps46257 ps46411 ps46527 ps46547 ps46550 ps46677 ps46746 ps46764 ps46765 ps46771 ps47000 ps47353 ps47411 ps47433 ps47593 ps47634 ps48770 ps48772 ps49101 ps49371 ps49385 ps50276 ps53358 ps53520 ps54103 ps54451 ps54474 ps54479 ps55305 ps55329 ps61039 ps61511 ps61604 ps62698 ps63780

Update Nov 13th 7:10pm PST: All machines are back online again, so if you’re still having trouble with your sites on a listed VPS, please contact support with the details as it’s unrelated. – Andrea

Previous updates available after the jump →


1:40 pm

Shared server ‘zaurak’ down due to file system check (RESOLVED)

Posted (November 11th, 2011 at 1:40 pm PST) by Oscar

The shared server ‘zaurak’ was having issues booting due to it’s raid card. On the final reboot it went into read only mode. Our admin team now has to perform a file system check on the machine. We are starting the file system check now while we prepare another machine to mount backups to. Once the backup machine is ready we will mount backups from zaurak so that your sites are running from backups while the fsck finishes on the old machine. We will update this post as soon as the backup machine is up.

UPDATE Nov 11th 11:55PM PST: The apache services on this server have all been up and running smoothly for the last 30 minutes and our admins report that all is well. This issue has been resolved now but if you have any problems with sites on this server please contact support. –JJ G Previous updates available after the jump →


9:08 am

DHApps and Easy One Click stability issues (RESOLVED)

Posted (November 11th, 2011 at 9:08 am PST) by mir

We are currently experiencing some issues with the servers which support our DHApps and Easy One Click installations. Our Admin team is working hard to resolve the matter but we do apologize for the inconvenience this is causing users of those two systems. We’ll provide more details as soon as they are available. -John

Update Nov 11th, 10:45am PST: This issue appears to be resolved – our admin team will continue to monitor the situation but you should see no further service disruption. Thank you for your patience! -John

Previous updates available after the jump →


11:43 am

WebFTP changes coming on Monday, November 14, 2011 11-11:30am PST (RESOLVED)

Posted (November 10th, 2011 at 11:43 am PST) by andrea

Good news! After a long and thorough search and testing period, we’re happy to report that we found a better webFTP software for your convenience. We’ll be removing the old (net2ftp) and installing the new (AjaXplorer: http://www.ajaxplorer.info ) on Monday, November 14, 2011, starting 11am PST.
Once this process starts, you won’t be able to use WebFTP until we’re done. We estimate this change will take no more than 30 minutes, but please make sure you’re not using webftp while we’re switching over to the new application.
All WebFTP references will remain on the DreamHost panel and will link to the new software afterwards.

Sorry about any inconvenience this may cause. Please direct any questions you may have about this change to support. -Andrea

Update Nov 14th 1:30pm PST:The upgrade has been successfully completed. The new application is now available for use through the DreamHost panel, under the usual WebFTP links. Please note, you’re currently unable to view hidden files (dotfiles) using this application, but we have plans to add that later. If you need any help, please contact support directly with the specifics. – Andrea


6:50 am

Shared webserver “majuro” unscheduled downtime (RESOLVED)

Posted (November 10th, 2011 at 6:50 am PST) by Justin@DH

The majuro webserver suddenly stopped responding a short time ago, and is not coming back up after a powercycle. Our admins are on the way to that specific datacenter to give the server a hands-on examination for what is causing the issues. Once they have found out a cause and have given us an ETA, we will post this here, as well as any other status updates in the meantime.

Update Nov 10th, 9:00am PST:The problem hardware has been replaced, and the admins have given us the all OK for this server. If you notice further issues and you are on this webserver, please contact support. -Justin

Update Nov 10th, 8:00am PST:The server is being investigated right now. It is definitely a hardware issue, and a few possible causes are being looked at. Worst case the admins expect to be able to swap the hard drives into a new chassis and get the server back up that way. ETA is currently around 1 hour unless the current hardware can be recovered. -Justin


12:31 am

Shared web server ‘tokyo’ offline for hard drive maintenance (RESOLVED)

Posted (November 10th, 2011 at 12:31 am PST) by patrickm

The shared web server ‘tokyo’ is offline at this time due to a bad drive which is causing boot issues for this server. Our engineers are working on the issue now and we expect to have this resolved shortly! We will update this post with any new details we can provide. Please contact Support with any questions!

Update Nov 10th, 1:23am PST: The hard drive was successfully replaced and the server is now back online! Please contact Support if you have any issues, or questions! -PatrickM


9:36 pm

Server ras-alhague is suffering from a DDOS attack. (RESOLVED)

Posted (November 9th, 2011 at 9:36 pm PST) by Brian@dreamhost

At this time, the server ras-alhague is suffering from a Denial of Service attack, causing the machine’s network interface to become saturation. This will make the machine appear down. We’re working on getting this mitigated, and hope to have it taken care of ASAP. We apologize for any inconvenience this may cause! We’ll update as things happen.

UPDATE Nov 10th 3:30AM PST: It looks like everything has been working well since our admins nullrouted the IP being attacked. The server has been fine for several hours at this point and this issue has been resolved. If you are still having any trouble please contact support. –JJ G Previous updates available after the jump →


5:09 pm

ISP downtime 11/09/2011, 00:01 – 04:00 PST (RESOLVED)

Posted (November 8th, 2011 at 5:09 pm PST) by andrea

One of our transit service providers announced maintenance for tomorrow early morning, 11/09/2011, 00:01 – 04:00 PST. They assured us that we won’t ‘feel a thing’. Even if we do, they are just one of our transit service providers between our data centers, so our redundant coverage will ensure that you feel even less. We’re posting about this as a precautionary measure, but are quite confident that this will not cause customer downtime. – Andrea

UPDATE Nov 9th 1:00PM PST: This finished on time last night, and as expected, this downtime didn’t affect our customers. – Andrea


 
© 1996-2011, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
11:25 am

Downtime on shared machine ‘Nair-Al-Saif ‘ ( RESOLVED)

Posted (November 8th, 2011 at 11:25 am PST) by brians

We are noticing intermittent load spikes on the machine. This would explain any failed connections that you are experiencing. All services ( HTTP, FTP, SSH, etc ) are impacted by this outage. Our admins believe that this is caused by a SYN Attack that was launched on the machine. They are diligently working towards having the machine operating at optimal performance. We sincerely apologize for any inconvenience this has caused. Your continued patience is greatly appreciated. Please check back for updates.

UPDATE Nov 8th 2:01PM PST: The attack has been blocked. Our admins decided that the best course of action would be to disable the target. Which happened to be user that was using their dreamhost account in a fraudulent manner.

At any event, all services on the machine have been restored. We sincerely apologize for any inconvenience this has caused. Thanks again for bearing with us. If you are still noticing problems, please submit a support request.

Previous updates available after the jump →


1:05 am

Shared server ‘guatemalacity’ offline for RAID card replacement (RESOLVED)

Posted (November 8th, 2011 at 1:05 am PST) by patrickm

We are sorry for the short notice, as we need to replace the RAID controller in the ‘guatemalacity’ web server.  The downtime from this should be less than 1 hour.  We will update this post when this is complete, or if there are any relevant updates during the process!  Please contact Support with any questions!

Update Nov 8th, 1:58am PST: This hardware maintenance is now complete! Please contact support with any issues!


10:41 am

Stability issues on shared machine ‘ Nitti ‘ ( RESOLVED )

Posted (November 7th, 2011 at 10:41 am PST) by brians

This would explain any type of lag, downtime that you are experiencing. Our admins are currently investigating load spikes on the machine. Unfortunately, these load average spikes are impacting all services ( HTTP, FTp, SSH, etc ). We are diligently working towards stabilizing the machine. Pinpointing the source has been rather difficult thus far. I will post more updates as information is made available. Sorry for any inconvenience this has caused. Your continued patience is greatly appreciated. Please check back for updates.

UPDATE Nov 7th 11:49AM PST: The load has been stabilized. It appears that our process monitor was not running. This allowed users to consume more than their fare share of system resources. Which caused a strain on the machine. We have restarted the process monitoring service and it’s running effectively. The load average on the machine is back to normal range. Again, we apologize for any inconvenience this has caused. If you are still noticing problems, please contact Tech Support. – Brian S.

Previous updates available after the jump →


2:06 am

Shared web server ‘castor’ offline for unscheduled RAID card replacement (RESOLVED)

Posted (November 7th, 2011 at 2:06 am PST) by patrickm

We apologize for the short notice and the inconvenience that this downtime may cause, as the RAID card in the shared web server ‘castor’ needs to be replaced.  This should be completed within the hour and we will update this post when we have more details, or the issue is resolved!  Please contact Support with any questions!

Update Nov 7th, 3:49am PST: This maintenance was completed successfully and the server has been online and running well! Please contact Support if you are experiencing any issues with your sites hosted on this server! -PatrickM


9:41 pm

Shared web server ‘astana’ offline for unscheduled RAID card replacement (RESOLVED)

Posted (November 6th, 2011 at 9:41 pm PST) by patrickm

We are taking down the Shared Web Hosting server ‘astana’ for a quick replacement of the RAID card.  The downtime for this should be minimal and we will update this post as soon as the maintenance is complete!

Update Nov 6th, 10:33pm PST: This maintenance is now complete and the server is back online! Please contact Support with any questions! -PatrickM


3:06 pm

aldib server down for RAID card replacement

Posted (November 6th, 2011 at 3:06 pm PST) by danwy

The ‘aldib’ server crashed and is not coming up after a reboot due to a RAID card failure. Our admin team is on their way to replace the RAID card and it should be back up shortly. We currently don’t have an ETA, but hope to have it back up within the hour. We’ll update this status post once we have more information on this server.

Update Nov 6, 4:17pm PDT: We are happy to report that the adib server is back online again. Sorry about the inconvenience the downtime has caused. If you’re still having issues with your domain on this server, it’s likely unrelated, so please contact support with the details. – Andrea

Previous updates available after the jump →


8:19 pm

Shared MySQL server ‘sandtiger’ downtime due to drive problems (RESOLVED)

Posted (November 5th, 2011 at 8:19 pm PST) by jj

The shared mysql server sandtiger went down earlier and, after an attempt to reboot the server, the drives were not recognized at startup. Our admins are looking into this issue and are going to swap the raid card in an attempt to get the server responsive once again.

We will be keeping you up to date as the issue is being worked on.

Update Nov 5, 9:24pm PDT: The server has been holding solid for the last half hour and it all seems to be working as expected. If you are using MySQL on sandtiger and have any problems please let support know. -JJ G Previous updates available after the jump →


2:32 am

Vserver host ‘homie-vserver141′ offline for unscheduled hardware swap (RESOLVED)

Posted (November 4th, 2011 at 2:32 am PST) by patrickm

We are sorry to report that the Vserver host ‘homie-vserver141′ is being taken offline for a full hardware swap, due to unexpected issues with the current hardware.  This process should take about 20-30 minutes and we will update this post when that is complete!  This will cause the following VPS guests to be offline during this time:

ps20361 ps20362 ps20375 ps20378 ps20384 ps20514 ps20780 ps20828 ps20977 ps21003 ps21012 ps21015 ps21316 ps21656 ps21765 ps21838 ps22041 ps22065 ps22551 ps23083 ps23084 ps23284 ps23310 ps24489 ps25476 ps25631 ps25942 ps29157 ps29458 ps30753 ps30980 ps31045 ps34234 ps34718 ps35128 ps37649 ps38001 ps38835 ps39769 ps42404 ps42413 ps43605 ps46926 ps47372 ps47663 ps47753 ps48084 ps54722 ps54734 ps54735 ps54848 ps54884

Please contact Support with any questions!

Update Nov 4th, 3:31am PDT: Our apologies for the delay in the resolution of this issue! The hardware swap has been completed and all services for this host are back online! Please contact Support if you notice any issues following this maintenance! -PatrickM


6:21 pm

Networking issues affecting some machines in our LAX datacenter (RESOLVED)

Posted (November 3rd, 2011 at 6:21 pm PST) by Oscar

We are currently experiencing networking issues with two racks at our LAX datacenter. Our admin team is on site and currently working on the issue. We will update this post as soon as we have more information regarding this outage. The affected machines are below:

Shared MySQL servers:

spurdog daggernose speartooth smoothhound lamnidae

Shared Hosting servers:

bertha oranjestad pagopago avarua stanley torshavn cayenne papeete gibraltar hagatna sainthelier funchal mamoudzou noumea

Web VPS Servers:

homie-vserver188
homie-vserver217

MySQL VPS servers:

homie-vsql51
homie-vsql52
homie-vsql48
homie-vsql50
homie-vsql57
homie-vsql49
homie-vsql53
homie-vsql54
homie-vsql55
homie-vsql56
homie-vsql60

This is affecting both shared and VPS servers. We will update this post as soon as we have more information.

Update Nov 3, 9:25pm PDT: We’ve received word from the admin working on this that the issue has been resolved as of about 30 minutes ago and that the switch is running without problems now. We were able to fix the problem with the existing switch without having to replace it (which would have taken longer to resolve the issue). There was a misconfiguration with the port channel on it which required it to be destroyed and rebuilt. We’re going to resolve this post now. If you have any further issues with your sites, please contact support so that they can investigate the issue further for you. Again, we apologize for the inconvenience this caused you and thank you for your patience as we worked on resolving it. -dan w

Previous updates available after the jump →


2:43 pm

homie-vserver147 down due to hardware issues (RESOLVED)

Posted (November 2nd, 2011 at 2:43 pm PST) by Oscar

The VPS host machine “homie-vserver147″ is currently down due issue’s with it’s hardware. We rebooted the machine a few times, but unfortunately it is still not responding correctly. Our admin team is troubleshooting this machine and we will update this post as soon as we have more information. The affected machines are:

ps33734 ps38398 ps38433 ps38450 ps38456 ps38474 ps38530 ps38538 ps38545 ps38587 ps38595 ps38637 ps38663 ps38882 ps39370 ps39374 ps39504 ps39548 ps39702 ps39951 ps39953 ps40005 ps40440 ps42493 ps43843 ps43859 ps44267  ps44829 ps45455 ps47444 ps47730 ps47829 ps48611 ps48616 ps50225 ps50598  ps52505 ps53156 ps54133 ps54375 ps54507 ps55081 ps55257 ps55347 ps55681

 

Update Nov 2nd, 4:10pm PDT: The machine is back up now. The networking cable for the machine went bad. We noticed this as there was no connectivity to the NIC on the server and when we replaced the cable it worked. My sincere apologies for the downtime and any inconvenience that this has caused. -Oscar

Update Nov 2nd, 3:10pm PDT: The machine is still not booting completely and is stuck in the boot process. Our admin team is on their way to data-center to work on the machine. We will update this post as soon as they are on site and we get more updates. -Oscar


 
© 1996-2011, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
12:11 pm

VPS host machine “Homie-vserver143″ offline. (RESOLVED)

Posted (November 2nd, 2011 at 12:11 pm PST) by Brandon

The VPS host machine “homie-vserver143″ has been taken offline in order to rectify some file system issues. We will update this post with further information as it becomes available.

Here is a list of affected VPS guests:

ps20371 ps20372 ps20405 ps20421 ps20574 ps20800 ps21032 ps21202 ps21554 ps21601 ps21620 ps22605 ps22615 ps23060 ps23305 ps23307 ps23986 ps24049 ps24150 ps24193 ps24405 ps24538 ps24619 ps24939 ps25285 ps25298 ps28738 ps28934 ps28969 ps29317 ps29633 ps31176 ps31179 ps37193 ps37207 ps37609 ps39644 ps43539 ps54085 ps54692 ps54712 ps57780 ps57936 ps65004

Update Nov 2nd, 12:26pm PDT: The machine is back online. We’ve replaced a bad drive in the machine and things are looking good. If you’re still experiencing any issues, please contact our Tech Support team. -Brandon


11:03 am

VPS Host machine “homie-vserver145″ experiencing intermittent downtime (RESOLVED)

Posted (November 2nd, 2011 at 11:03 am PST) by Brandon

The server appears to be mounting the file system read only, which may mean that it needs RAID hardware replaced. We’re currently isolating the cause of the issue and will update this post as soon as we have more specific information.

All affected VPS guests are listed below:

ps20357 ps31241 ps34608 ps34609 ps38422 ps38466 ps38500 ps38511 ps38611 ps38619 ps38669 ps38675 ps38682 ps39235 ps39458 ps39582 ps39600 ps39725 ps39890 ps40352 ps40796 ps41245 ps42511 ps42516 ps43930 ps44022 ps44204 ps44269 ps44271 ps44369 ps45849 ps46100 ps46226 ps46275 ps46458 ps47479 ps48143 ps48327 ps48559 ps50448 ps52292 ps52371 ps53130 ps53135 ps53589 ps54079 ps54258 ps54379

Update Nov 2nd, 2:00pm PDT:   All VPS machines are now back up and working on homie-vserver145. If you have any issues with your VPS being down and it was on the list for this post, please contact our support team immediately and we will be glad to look into the issue for you. -Oscar

Previous updates available after the jump →


1:52 am

Networking issue between two of our datacenters (RESOLVED)

Posted (November 2nd, 2011 at 1:52 am PST) by patrickm

We are currently investigating a networking issue between our LAX and Irvine data centers. This issue is causing connection issues between a handful of Web VPSs and their MySQL counterparts. As this is not affecting a specific group of servers, we are not able to provide a list of affected customers at this time. Our Admins are working on resolving this issue as quickly as possible.

We will update this post with more details as soon as possible! Please contact Support with any questions!

Update Nov 2nd, 11:50am PDT: Sorry for the late update! This issue was permanently resolved just before 10am today. While the post was marked resolved, it seems we didn’t provide an update with this information. -Brandon

Previous updates available after the jump →


4:14 am

dreamhost.com offline (RESOLVED)

Posted (November 1st, 2011 at 4:14 am PST) by patrickm

Our apologies for the downtime with our main website, dreamhost.com.  An issue with the back-end configuration of the site is keeping it offline at this time.  We are working to resolve this issue now, but this should not affect any hosted services.  You can still access your Web Panel with the following link:

https://panel.dreamhost.com

Please feel free to contact Support from your Web Panel with any questions! We will update this post shortly!

Update Nov. 1st, 4:32am PDT: This issue is now resolved! Our trusty Admins were able to get this working again and we will be making sure that this issue is investigated and resolved permanently! -PatrickM


1:51 pm

Downtime on Homie-Vsql57 (RESOLVED)

Posted (October 31st, 2011 at 1:51 pm PST) by brians

The machine is not responding to remote reboots. We have requested that one of admins at the data center try a manual reboot. Below are of list of PS-mysql guest that are impacted by this outage. We sincerely apologize for any inconvenience this has caused and appreciate your continued patience. Please check back for updates.

psmysql30850 psmysql30908 psmysql30910 psmysql30937 psmysql30963 psmysql30979 psmysql30983 psmysql30986 psmysql30990 psmysql30992 psmysql31008 psmysql31010 psmysql31058 psmysql31536 psmysql33096 psmysql33387 psmysql33600 psmysql33803 psmysql33836 psmysql33929 psmysql33940 psmysql34000 psmysql34799 psmysql35539 psmysql35673 psmysql35764 psmysql36277 psmysql36330 psmysql39333 psmysql40247 psmysql44088 psmysql44170 psmysql46423 psmysql47141 psmysql47526 psmysql47529 psmysql47544 psmysql48176

 

Update Oct 31st, 2:10pm PDT:The server is back online and all MySQL services are up and running. Should you still be experiencing any issues, please contact our Technical Support department. -Brandon


12:45 am

homie-vserver200 offline for unscheduled file-system check (RESOLVED)

Posted (October 31st, 2011 at 12:45 am PST) by patrickm

We apologize for the downtime, as the VPS host server ‘homie-vserver200′ is offline for an unscheduled file-system integrity check.  We expect this process to take a few hours and will affect the following VPS guests:

ps40135 ps40154 ps40194 ps40228 ps40242 ps40299 ps40302 ps40365 ps40469 ps40679 ps40814 ps41123 ps41522 ps41619 ps41625 ps41877 ps41927 ps42061 ps42423 ps43643 ps44102 ps44437 ps44522 ps44866 ps45207 ps46214 ps47296 ps48025 ps48116 ps48127 ps48905 ps49580 ps50187 ps50219 ps50254 ps50330 ps51763 ps51780 ps52411 ps52545 ps52937 ps53315 ps53351 ps53366 ps53393 ps53448 ps53588 ps53968 ps54247 ps54406 ps55246 ps55325 ps55331 ps55638 ps55651 ps55659 ps55694 ps55711 ps61595 ps62793

Update Nov. 1st, 9:30am PDT: The admin team has notified us that the restore has completed. All VPS should be back up and working again, using their most recently created backups. If you notice any issues with your VPS not working as expected, please contact support. -Justin

Previous updates available after the jump →


4:34 pm

Downtime on shared web server Zaurak – file system check (RESOLVED)

Posted (October 29th, 2011 at 4:34 pm PST) by andrea

We had to reboot your shared server ‘zaurak’ and when it came back, the partition where the home directories are stored didn’t mount to the machine and the apache service wouldn’t start. We started a file system check to find out why and to ensure data integrity. We’re also working on setting up a backup machine, so your sites would be up and running while we conduct the file system check. I’ll update this post when the backup machine is up. Sorry about the inconvenience this is causing. – Andrea

UPDATE Oct 30th 6:40AM PDT:The file system check has finished, the original hardware is live again like previously setup. If you notice any further issues please let us know.

Previous updates available after the jump →


9:52 am

Shared server Managua being failed over to new hardware ( RESOLVED )

Posted (October 28th, 2011 at 9:52 am PST) by brians

This would explain the downtime that you are experiencing at the moment. All services http, ftp, etc are affected by this outage. Our admins plan to move the machine to new hardware and create a NFS mount to the machine that hosts your backup data. This will allow your sites to remain online as the FSCHK is being performed on the old hardware. I will update this post once the new hardware is deployed and the NFS mount has been created.

We sincerely apologize for any inconvenience this has caused. Your continued patience is greatly appreciate. Please check back for updates.
Updated information: Apache services are all up and running while the File System Check proceeds.

Update Oct 29th, 10:56pm PDT: The restore has completed. – Sandon V.

Update Oct 29th, 6:30pm PDT:The restore is almost finished. We’ll update this post again when it’s all done. We appreciate your patience! – Andrea

Previous updates available after the jump →


6:30 pm

Irvine Datacenter Networking Issues (RESOLVED)

Posted (October 26th, 2011 at 6:30 pm PST) by Oscar

We are currently experiencing networking issues with one of the server rows which is causing downtime for some dedicated customers. Our admins are currently working on the issue and we will update this post as soon as we have more information. The affected machines are:

ds2103 ds1912 ds1892 ds1891 ds1890 ds1889 ds1888 ds1886 ds1885
ds1898 ds1897 ds1896 ds1895 ds2091 ds1893 ds1899 ds1900 ds1901
ds2131 ds1904 ds1905 ds1906 ds1907 ds1908 ds1909 ds1910 ds1914
ds1915 ds1916 ds1917 ds1918 ds1919 ds2132 ds1921 ds1922
ds2101 ds2090 ds1933 ds1950 ds1949 ds1948 ds1947 ds1946 ds1945
ds1943 ds1944 ds1942 ds1951 ds1952 ds1953 ds1955 ds1957 ds1958
ds1959 ds1960 ds1941 ds1940 ds1938 ds1939 ds1936 ds1937 ds1934
ds1935 ds1932 ds1926 ds1927 ds1928 ds1929 ds1930 ds1931
ds2062 ds2063 ds2064 ds2043 ds2044 ds2037 ds2036 ds2042 ds2041
ds2029 ds2030 ds2040 ds2027 ds2028 ds2035 ds2031 ds2045 ds2046
ds2038 ds2039 ds2033 ds2034 ds2053 ds2047 ds2048 ds2049 ds2052
ds2051 ds2050 ds2061 ds2060 ds2059 ds2058 ds2057 ds2056 ds2055
ds2032 ds2054

ds2151 ds2152 ds2153 ds2154 ds2155 ds2134 ds2135 ds2156 ds2136
ds2137 ds2157 ds2138 ds2139 ds2140 ds2141 ds2142 ds2143 ds2144
ds2145 ds2146 ds2147 ds2148 ds2149 ds2150 ds2097 ds2098 ds2099
ds2100 ds2086 ds2087 ds2088 ds2089 ds2102

Update Oct 26th,  7:45pm PDT:  This is completely resolved now all machines listed in this post are now back up. Should you have any further issues with your sites being down, please contact our support team immediately and we will be glad to look  it for you. -Oscar

Update Oct 26th,  6:45pm PDT:  Our network tech is on route to the datacenter. Once there, we will update this post once we have more information for you. -Oscar


12:58 pm

Panel Downtime (RESOLVED)

Posted (October 25th, 2011 at 12:58 pm PST) by brians

We are fully aware of the problems encountered when attempting to access our Web Panel http://panel.dreamhost.com. A member of our administration team is currently investigating the matter. We will post updates as they are made available. Your continued patience is greatly appreciated. Please check back for updates.

Update Oct 25th 2:20pm PDT: The issues with the web panel slowness have been completely resolved now. We initially thought we had things working better about two hours ago. Unfortunately the slowness issues came back. The good news is that this time we were able to track down the main cause of the slowness and correct it. -Oscar


 
© 1996-2011, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
6:59 am

Shared server “cort” is offline due to file system check (RESOLVED)

Posted (October 25th, 2011 at 6:59 am PST) by Justin@DH

The shared web hosting server ‘cort’ has been taken offline for an unscheduled file-system check. The server rebooted to a read-only file-system, which is a sign of issues with the drives, so this check is being performed to verify the integrity of the data on the server. We will update this post as this process continues and once we have an ETA.

As of 9am, the NOC team has put up a temporary new cort server, which will host your sites from the backups we have. This should keep the sites running until the file system check completes on the current cort server. Apaches are coming up and sites should be also shortly.

Update Oct 26th 7:55pm PDT: The filesystem check has completed and we have moved over back to the original machine. All of your sites are now back on the original machine. Should you have any issues with your sites now, please contact our support team and we will be glad to help you out. -Oscar

Previous updates available after the jump →


1:08 am

Operating System Upgrade for 30 machines on Oct. 25th at 10pm PDT – machine names in post (RESOLVED)

Posted (October 25th, 2011 at 1:08 am PST) by patrickm

We will continue the OS upgrades on our shared hosting machines on Monday Oct. 24th at 10pm.

We would like to also bring the following to your attention, which will help you with getting resolutions to any Rails issues that may come up during this process:

1. Upgrade to mysql2 gem is highly suggested (and database.yml updated)
You can use either mysql version 2.8 or mysql2. The reason for this is that mysql libraries changed in squeeze and both rely on specific versions.

2. You should have GEM_HOME and GEM_PATH set in your gemrc (and if you so desire, in your environment.rb file). We need to maintain that our customers manage their environment variables as the Rails community highly suggests.

3. You should all be installing gems locally and should (start to) use bundler as the locking method to lock down your gem installs to vendor/gems (http://gembundler.com/)

Each machine will be down for roughly one hour during the upgrade. We apologize for downtime caused by this upcoming upgrade. On the night of the upgrades we will keep the status post updated with information on which machines are started and which machines have completed. The following machines will be upgraded on Oct. 24th:

krank kraz ksora kualalumpur kuma kustom kuwaitcity kyiv lacerta laney lansky lapaz leah legs lepke lepus lesath liberty libra libreville lightning lilongwe lima line6 lion-o lisbon lizards ljubljana lome london

To find out if your server is affected in this batch, go to the panel and check the machine listed for your FTP/Shell user.

Please note, under no circumstance will we start the upgrades before the scheduled time, so if you’re having any problems that isn’t during the upgrade on the listed machines, it’s unrelated. Please contact support with the details, so we can assist you.

Update Oct 26th 2:20am PDT: The previous list of servers have been successfully updated. This covers the entire list for the night. If you have any problems on any of these servers please let support know. –JJ G Previous updates available after the jump →


9:46 pm

Operating System Upgrade for 32 machines on Oct. 24th at 10pm PDT – machine names in post (RESOLVED)

Posted (October 23rd, 2011 at 9:46 pm PST) by Jason C

We will continue the OS upgrades on our shared hosting machines on Monday Oct. 24th at 7pm 10pm. We have worked out some of the issues that occurred in our previous upgrade attempts.

We would like to also bring the following to your attention, which will help you with getting resolutions to any Rails issues that may come up during this process:

1. Upgrade to mysql2 gem is highly suggested (and database.yml updated)
You can use either mysql version 2.8 or mysql2. The reason for this is that mysql libraries changed in squeeze and both rely on specific versions.

2. You should have GEM_HOME and GEM_PATH set in your gemrc (and if you so desire, in your environment.rb file). We need to maintain that our customers manage their environment variables as the Rails community highly suggests.

3. You should all be installing gems locally and should (start to) use bundler as the locking method to lock down your gem installs to vendor/gems (http://gembundler.com/)

Each machine will be down for roughly one hour during the upgrade. We apologize for downtime caused by this upcoming upgrade. On the night of the upgrades we will keep the status post updated with information on which machines are started and which machines have completed. The following machines will be upgraded on Oct. 24th:

jaguars jakarta jayturser jazz jbplayer jerusalem jets jih johnnyblaze johnson joker jubilee kabul kakuji kalamazoo kampala karpis kathmandu kay keid khartoum kigali kingpin kings kingston kingstown kinshasa kleeia knicks kochab kornephoros kramer

To find out if your server is affected in this batch, go to the panel and check the machine listed for your FTP/Shell user.

Please note, under no circumstance will we start the upgrades before the scheduled time, so if you’re having any problems that isn’t during the upgrade on the listed machines, it’s unrelated. Please contact support with the details, so we can assist you.

Update Oct 25th 4:45am PDT: All upgrades for tonight are complete! Servers and services are back online, so please contact Support if you have any issues! -PatrickM

Previous updates available after the jump →


8:43 am

Vserver host machine ‘homie-vserver25′ down (RESOLVED)

Posted (October 22nd, 2011 at 8:43 am PST) by Oscar

The vserver host machine ‘homie-vserver25′ is currently offline due to experiencing hardware issues. Our admins are currently working on a resolution, and we will get things back online as quickly as possible.

Here is a list of affected VPS:

ps10037 ps10941 ps17558 ps17559 ps17561 ps17562 ps18199 ps19572 ps19614 ps19668 ps26301 ps26503 ps27037 ps27044 ps30777 ps39340 ps39382 ps39384 ps39386 ps39586 ps39635 ps39866 ps39940 ps46444 ps46448 ps47486 ps48158 ps48477 ps49162 ps49526 ps50460 ps50555 ps52172 ps60825 ps60826 ps60827 ps60829 ps62579 ps6636 ps6641 ps6674 ps6789 ps6792 ps6793 ps6794 ps6972 ps6974 ps7159 ps7184 ps7357 ps7398 ps7466 ps7514 ps7731 ps7790 ps7826 ps7881

 We will update this post with any relevant information as it becomes available.

Update Oct 22, 9:10am PDT: The host machine is back up and all VPS guests are back online. Our sincerest apologies for any inconvenience the downtime has caused. If you are still having problems, please submit a support request and we will be glad to look into the issue for you.  -Oscar


9:40 am

Vserver host machine ‘homie-vserver225′ offline due to hardware issues ( RESOLVED )

Posted (October 21st, 2011 at 9:40 am PST) by Brandon

The vserver host machine ‘homie-vserver225′ is currently offline due to experiencing hardware issues. Our admins are currently working on a resolution, and we will get things back online as quickly as possible.

 

Here is a list of affected VPS:

 

 ps25286 ps37472 ps47015 ps47024 ps47025 ps47026 ps47035 ps47037 ps47038 ps47039 ps47055 ps47068 ps47069 ps47077 ps47095 ps47128 ps47143 ps47150 ps47171 ps47237 ps47252 ps47272 ps47365 ps47384 ps47393 ps47395 ps47430 ps47464 ps47500 ps47591 ps47669 ps47747 ps47752 ps47755 ps47791 ps47865 ps47869 ps47875 ps47896 ps48139 ps48192 ps48202 ps48203 ps48241 ps48288 ps48316 ps48325 ps48470 ps48732 ps48841 ps48847 ps49156 ps49337 ps49503 ps49643 ps49827 ps50119 ps50290 ps50623 ps52252 ps54138 ps59640 ps63934

 

We will update this post with any relevant information as it becomes available.

Update Oct 21st, 12:46pm PDT: All VPS guests are back online. Sorry for the inconvenience the downtime had caused. If you are still having problems, please submit a support request. It’s likely that the downtime is not a result of this reported ( RESOLVED ) outage. -Brian S

Previous updates available after the jump →


9:05 am

Shared server “castellano” is offline due to file system check ( RESOLVED )

Posted (October 20th, 2011 at 9:05 am PST) by mir

This server was rejecting I/O and on a reboot it is undergoing a file system check which should be completed shortly. Once that’s done we’ll be looking into the situation and will provide further details as they become available. -John

Update Oct 21st, 2:32pm PDT: The data has been successfully copied from the new hardware. If you you are still missing data, please contact Tech Support. We will gladly investigate further – Brian S

Previous updates available after the jump →


2:57 pm

Network Maintenance Thursday Night, October 20th, 10pm – 5am PDT ( RESOLVED )

Posted (October 18th, 2011 at 2:57 pm PST) by Brandon

We’ll be performing some scheduled network maintenance Thursday October 20th, between 10PM and 2AM. There may be some intermittent networking issues up to 30 minutes each as we upgrade hardware and power cycle various parts of our backbone, so please plan accordingly.

Please contact support should you experience any unusual issues stemming from this maintenance.

More information: This maintenance had more issues than expected resulting in longer services outages including downtime for DreamHost.com (which is back up) and servers throughout our network (we’re continuing to update this posting as we resolve those issues). One of the primary issues of MySQL connectivity has also been corrected. Dedicated server customers should now see their networking fixed.

For the record, we really do have _everyone_ working on this (all of our executive and management team members are up and involved in setting things right).

Update Oct 21th, 12:58pm PDT: The newtwork outage has been resolved. As previously mentioned, there were a few subnets that required more attention. Everything is back to normal. Again, we sincerely apologize for any inconvenience this has caused. Your patience is greatly appreciated. If you are still having problems, please submit a support request. It’s likely that your issue is not related to this status post. – Brian S

Previous updates available after the jump →


1:45 pm

Operating System Upgrade for 39 machines on Oct. 19th 7pm PDT – machine names in post (RESOLVED)

Posted (October 17th, 2011 at 1:45 pm PST) by andrea

We will continue the OS upgrades on our shared hosting machines on Tuesday Oct. 19th at 7pm. We have worked out some of the issues that occurred in our previous upgrade attempts.

We would like to also bring the following to your attention, which will help you with getting resolutions to any Rails issues that may come up during this process:

1. Upgrade to mysql2 gem is highly suggested (and database.yml updated)
You can use either mysql version 2.8 or mysql2. The reason for this is that mysql libraries changed in squeeze and both rely on specific versions.

2. You should have GEM_HOME and GEM_PATH set in your gemrc (and if you so desire, in your environment.rb file). We need to maintain that our customers manage their environment variables as the Rails community highly suggests.

3. You should all be installing gems locally and should (start to) use bundler as the locking method to lock down your gem installs to vendor/gems (http://gembundler.com/)

Each machine will be down for roughly one hour during the upgrade. We apologize for downtime caused by this upcoming upgrade. On the night of the upgrades we will keep the status post updated with information on which machines are started and which machines have completed. The following machines will be upgraded on Oct. 19th:

ulaanbaatar ursamajor ursaminor vaduz valletta valleyarts vaticancity vela venom victoria vienna vientiane vikings vilnius virgo voodoo vox shaula sheliak sheratan shock siegel silversurfer singapore sinistra situla skat skopje slippy slithe snowbird snowman sofia soldano sox sparks spawn spectre spica spiderman spillane spilotro spurs squeezie squier stampeders stanley starfire starfox steelers steinberger sterope-ii stjohns stockholm storm sualocin

To find out if your server is affected in this batch, go to the panel and check the machine listed for your FTP/Shell user.

Please note, under no circumstance will we start the upgrades before the scheduled time, so if you’re having any problems that isn’t during the upgrade on the listed machines, it’s unrelated. Please contact support with the details, so we can assist you.

Update Oct 20th, 2:25am PST: The final batch of servers for tonight are upgraded! We are pushing off the upgrade for the ‘ulaanbaatar’ server, as well. Please contact Support if you have any issues following these upgrades! -PatrickM

Previous updates available after the jump →


1:43 pm

Operating System Upgrade for 32 machines on Oct. 18th 10pm PDT – machine names in post (RESOLVED)

Posted (October 17th, 2011 at 1:43 pm PST) by andrea

We will continue the OS upgrades on our shared hosting machines on Tuesday Oct. 18th at 10pm. We have worked out some of the issues that occurred in our previous upgrade attempts.

We would like to also bring the following to your attention, which will help you with getting resolutions to any Rails issues that may come up during this process:

1. Upgrade to mysql2 gem is highly suggested (and database.yml updated)
You can use either mysql version 2.8 or mysql2. The reason for this is that mysql libraries changed in squeeze and both rely on specific versions.

2. You should have GEM_HOME and GEM_PATH set in your gemrc (and if you so desire, in your environment.rb file). We need to maintain that our customers manage their environment variables as the Rails community highly suggests.

3. You should all be installing gems locally and should (start to) use bundler as the locking method to lock down your gem installs to vendor/gems (http://gembundler.com/)

Each machine will be down for roughly one hour during the upgrade. We apologize for downtime caused by this upcoming upgrade. On the night of the upgrades we will keep the status post updated with information on which machines are started and which machines have completed. The following machines will be upgraded on Oct. 17th:

subra sugeknight suhail sulafat sunn suns sunspot supergirl suva swampthing syrma tabbuts tabit taipei talitha-australis ulaanbaatar ursamajor ursaminor vaduz valletta valleyarts vaticancity vela venom victoria vienna vientiane vikings vilnius virgo voodoo vox

To find out if your server is affected in this batch, go to the panel and check the machine listed for your FTP/Shell user.

Please note, under no circumstance will we start the upgrades before the scheduled time, so if you’re having any problems that isn’t during the upgrade on the listed machines, it’s unrelated. Please contact support with the details, so we can assist you.

Update Oct 19th, 6:11pm PST: Please note, the following machines will be upgraded tonight, as we weren’t able to complete the list last night.
ulaanbaatar ursamajor ursaminor vaduz valletta valleyarts vaticancity vela venom victoria vienna vientiane vikings vilnius virgo voodoo vox

Please keep an eye on Operating System Upgrade for 39 machines on Oct. 19th 7pm PDT – machine names in post for updates. – Andrea

Previous updates available after the jump →


1:41 pm

Operating System Upgrade for 36 machines on Oct. 17th 7pm PDT – machine names in post (RESOLVED)

Posted (October 17th, 2011 at 1:41 pm PST) by andrea

We will resume OS upgrades on our shared hosting machines on Monday Oct. 17th at 7pm. We have worked out some of the issues that occurred in our previous upgrade attempts.

We would like to also bring the following to your attention, which will help you with getting resolutions to any Rails issues that may come up during this process:

1. Upgrade to mysql2 gem is highly suggested (and database.yml updated)
You can use either mysql version 2.8 or mysql2. The reason for this is that mysql libraries changed in squeeze and both rely on specific versions.

2. You should have GEM_HOME and GEM_PATH set in your gemrc (and if you so desire, in your environment.rb file). We need to maintain that our customers manage their environment variables as the Rails community highly suggests.

3. You should all be installing gems locally and should (start to) use bundler as the locking method to lock down your gem installs to vendor/gems (http://gembundler.com/)

Each machine will be down for roughly one hour during the upgrade. We apologize for downtime caused by this upcoming upgrade. On the night of the upgrades we will keep the status post updated with information on which machines are started and which machines have completed. The following machines will be upgraded on Oct. 17th:

vultureman walsh waratahs wario warmachine warmouth warriors warsaw warwick washburn washingtondc wasp watanabe wellington willa windhoek wizards wizz-ra wolfsbane wollows wolverine wonderwoman yamaha yamoussoukro yangon yankees yaounde yerevan yorkville zagreb zaniah zatara zaurak zon zwerbach

To find out if your server is affected in this batch, go to the panel and check the machine listed for your FTP/Shell user.

Please note, under no circumstance will we start the upgrades before the scheduled time, so if you’re having any problems that isn’t during the upgrade on the listed machines, it’s unrelated. Please contact support with the details, so we can assist you.

Update Oct 17th, 10:57pm PST: The last of the upgrades for tonight have completed! Please contact Support if you notice any issues, or have any questions! -PatrickM

Previous updates available after the jump →


 
© 1996-2011, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
1:10 pm

Phpbb easy one-clicks having problems

Posted (October 16th, 2011 at 1:10 pm PST) by danwy

We’re currently experiencing some issues with our phpBB one click installs, and they are currently offline. Our admin team isn’t sure what the problem is yet, but they are investigating it now and hope to have it resolved ASAP. We don’t have many details at this time, but we will update this status post as soon as we have more information.

Update Oct 15th, 7:30am PST: We’ve resolved the issues with phpBB easy one clicks. It looks like there were several forums used primarily for spamming that were jacking up the load. Once we disabled them, the server loads settled back down and began to work normally again. We’re going to continue to find instances of easy phpBB one click installs used for spamming and will disable those as well. Sorry for any inconvenience this caused any of our customers.


5:29 pm

VPS host machine “homie-vserver124″ down for unscheduled maintenance (RESOLVED)

Posted (October 13th, 2011 at 5:29 pm PST) by Oscar

The VPS host machine “homie-vserver124″ is currently down due to emergency maintenance. Here is a list of affected VPS guests:

ps19325 ps19388 ps19427 ps19438 ps19468 ps19469 ps19476 ps19492 ps19531 ps19550 ps19589 ps19683 ps19971 ps19973 ps20056 ps20173 ps20174 ps20259 ps20367 ps20833 ps21064 ps21079 ps21543 ps21980 ps22751 ps22995 ps24038 ps24797 ps28657 ps28668 ps28758 ps28779 ps28905 ps29009 ps29546 ps30113 ps30308 ps30827 ps30958 ps30995 ps32946 ps36932 ps37294 ps37441 ps37630 ps37843 ps43286 ps43626 ps43856 ps48054 ps48058 ps52429 ps54637 ps54641

There is currently an issue with one of the disk drives which our admin team is investigating. We will keep this post updated with new information as it becomes available.

Update Oct 13th, 5:45pm PST:  This is now resolved and all VPS guests are now back up and working. Should you notice your VPS is down, please contact our support staff and we will be glad to look into the issue for you. -Oscar


9:08 pm

VPS host machine “homie-vserver157″ down for unscheduled file system check (RESOLVED)

Posted (October 10th, 2011 at 9:08 pm PST) by mir

The RAID card died on this server so after a quick replace and reboot it should have come right back up but required a file system check. The check has completed and it is rebooting now so this should be resolved momentarily. These are the VPS guests which are on that host:

ps23329 ps23362 ps23551 ps23906 ps24118 ps24207 ps24250 ps24292 ps24440 ps24464 ps24497 ps24535 ps24730 ps24957 ps25038 ps25209 ps28480 ps28812 ps28907 ps28914 ps29028 ps29064 ps29198 ps29278 ps29442 ps29619 ps29651 ps29996 ps30982 ps32106 ps34508 ps34689 ps36254 ps37250 ps37712 ps43875 ps44572 ps48338 ps48431

Update Oct 10th, 9:10pm PST: As it turns out everything is coming back online correctly. We thank you for your patience. – John


1:33 pm

‘Lakers’ server having some problems (RESOLVED)

Posted (October 10th, 2011 at 1:33 pm PST) by danwy

The server ‘lakers’ is currently having some issues and is hanging on reboot. We ran a fsck on it and it came up error free, but it is still hanging on reboot. One of our admins is currently working on it and hopes to have it back up and running ASAP. We’ll update this status post shortly once we have more information.

Update Oct 10th, 2:10pm PST: The server is on its way back up now after a RAID card replacement. We’re going to make sure that everything comes up smoothly and will resolve this status post shortly if everything goes smoothly. Keep your eyes on this post for updates. -dan w

Update Oct 10th, 2:30pm PST: The server is back up and running now. There was an issue with the root filesystem being located in the wrong location. We’re not sure how that happened, but we will be investigating the cause further. If you notice any further issues with a site hosted on the ‘lakers’ server, please contact support right away. -dan w


9:25 am

Shared server ” acrab ” is offline due to RAID card replacement ( RESOLVED )

Posted (October 10th, 2011 at 9:25 am PST) by brians

The shared server “acrab” has a faulty RAID card. We’ve decided that the best course of action would be to replace the RAID card. A member of our admin team is en route to the data center as I type. All services ( http, ftp, etc ) will remain offline until the RAID has been replaced. We sincerely apologize for any inconvenience this has caused and appreciate your patience. Please check back for future updates.

UPDATE Oct 10th 11:01 AM PST: The RAID card has been replaced and the machine is back online. All services are up and running. Sorry for any inconvenience this has caused. If you are still having problems, please submit a support request.


6:30 am

Shared server “penguins” down for unscheduled file-system check (RESOLVED)

Posted (October 10th, 2011 at 6:30 am PST) by Justin@DH

The shared server penguins was undergoing a file system check but now we’ve progressed from that to failing the machine over to new hardware. We’re serving sites via backup as we copy that data over to the new hardware but you’ll be able to make changes to your data and access your site during the restore process though performance may be degraded at times (due to the copying data). We’ve also been able to get access to the old hardware’s data now so if more recent data than the backup from the 8th (which is currently being used) is needed please contact support.

Update Oct 16th, 7:38pm PST: Our admins have notified us that the server has finished its restore process and has been flipped over to the new hardware. It’s been running without issue for a little while now, but we’ll continue to monitor things. If you have a site on penguins and happen to notice any problems, please contact our support team as soon as possible. – Jason Previous updates available after the jump →


4:24 pm

Operating System Upgrade for 30 machines on Oct. 11th 7pm PDT – machine names in post (RESOLVED)

Posted (October 7th, 2011 at 4:24 pm PST) by Ralph

We will resume OS upgrades on our shared hosting machines on Tuesday Oct. 11th at 7pm. We have worked out some of the issues that occurred in our previous attempt upgrading 50 machines on Oct. 3rd. We apologize for downtime caused by this upgrade. Each machine will be down for roughly 1 hour during the upgrade. On the night of the upgrades we will keep the status post updated with information on which machines are started and which machines have completed. The following machines will be upgraded on Oct. 11th:

galactus galaxy gambino gambit gemini genovese georgetown ghostrider giancana gianfar gibraltar giedi godin gomeisa gotti graffias gravano greene greengoblin greenlantern gretsch grizzlies grumium grus gtx guatemalacity guild hachiman hadar haddock

Update Oct 12th, 12:26am PST: The upgrades for tonight are all done! Please contact Support if you encounter any issues! -PatrickM

Previous updates available after the jump →


10:43 am

Shared Server “bolkens” offline after RAID card replacement. (RESOLVED)

Posted (October 7th, 2011 at 10:43 am PST) by Brandon

The shared server “bolkens” has been taken offline in order to replace a faulty RAID card, but has not come back online properly. Our admin team are diligently working on a resolution, and we will provide further information as it becomes available.

 

Update Oct 7th, 11:17am PST:  The “bolkens” webserver is now coming back online, the issue booting seems to have been transient, and Apache services are up and running. If you’re still seeing any issues, please contact our helpful support staff.


1:52 pm

VPS host machine “homie-vserver143″ offline, being moved to new hardware. (RESOLVED)

Posted (October 6th, 2011 at 1:52 pm PST) by Brandon

The VPS host machine “homie-vserver143″ is being taken offline in order to replace faulty hardware. Here is a list of affected VPS guests:

ps20371 ps20372 ps20405 ps20421 ps20574 ps20800 ps21032 ps21172 ps21202 ps21554 ps21601 ps21620 ps22605 ps22615 ps23060 ps23305 ps23307 ps23986 ps24049 ps24150 ps24193 ps24405 ps24538 ps24619 ps24939 ps25285 ps25298 ps28738 ps28934 ps28969 ps29317 ps29633 ps31176 ps31179 ps37193 ps37207 ps37609 ps39644 ps43539 ps54085 ps54692 ps54712 ps57780 ps57936

We will keep this post updated with new information as it becomes available.

Update Oct 6th, 4:28pm PST: Good news everyone! One of our team members managed to recover the RAID array after all. The host server and all VPS customers on it should be back up as of now. If you happen to notice any issues with your sites or VPS servers, please contact our support team immediately. – Jason Previous updates available after the jump →


5:11 am

Shared server ‘suva’, down for unscheduled file-system check ( RESOLVED )

Posted (October 6th, 2011 at 5:11 am PST) by jj

There was an issue with a reboot of the shared server suva, which is requiring that our admins reimage the OS partition of the drive, causing downtime on this machine. We’ll update this post as soon as more information becomes available.

Unfortunately the server is now in need of a file system check, updates will be posted on the progress of that check as it runs.

Update Oct 6th, 3:14pm PST: All apache’s have been back up and running for over an hour. We’ve been monitoring the situation since then and things have been running smoothly, so we’re going to resolve this post now. If you continue to have issues with any sites on this server, please contact support so that they can investigate it further. -dan w

Previous updates available after the jump →


 
© 1996-2011, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
1:39 am

Shared server ‘peacock’ is down (RESOLVED)

Posted (October 6th, 2011 at 1:39 am PST) by jj

Our admins are currently looking into the issue as to why the shared server peacock is currently down and non-responsive, currently it looks like a hardware related network issue. This has caused a general outage for any customer hosted on that machine, we will be updating this post with more information as soon as we have it.

Update Oct 6th, 2:15am PST: The problem here was a bad network cable, our admins have replaced it and the machine has been running smoothly for the last half hour. If you still have any problems please let support know. -JJ G


7:43 am

Shared server “alaraph” offline for a RAID card replacement (RESOLVED)

Posted (October 5th, 2011 at 7:43 am PST) by Justin@DH

The shared server alaraph is having issues with its raid card and our datacenter team is working on replacing it. We hope to have the server back up shortly. When we have further information about the status, an ETA or it is fixed, we will update this post.

Update Oct. 5, 9:01am PST: The raid card has been replaced and the server is back up. If you notice any further issues, please contact support. – Justin


3:50 am

Shared server ‘penguins’ offline for unscheduled file-system check (RESOLVED)

Posted (October 5th, 2011 at 3:50 am PST) by patrickm

The shared server ‘penguins’ came back from a reboot with a read-only file-system which means we must complete a file-system check to verify the integrity of the data on the drives.  We expect this process to take a number of hours to complete and we will update this post as that progresses!  Please contact Support with any questions you might have!

Update Oct 5th, 4:35am PST: Great news! The file-system check finished quickly and came back without any errors! The server is back online! Please contact Support with any questions! -PatrickM


4:20 pm

WebFTP temporarily disabled (RESOLVED)

Posted (October 4th, 2011 at 4:20 pm PST) by Oscar

WebFTP has been disabled to fix up a security vulnerability that has been found in it. We will re-enable this service once this issue has been addressed. Unfortunately there is no ETA for when this will be resolved. For the time being, please make sure to use an alternate FTP client in order to log into the server. You can find alternate FTP software recommendations in our wiki page here:

http://wiki.dreamhost.com/FTP#FTP_Client_Software

Update – Oct 15th: We still don’t have an ETA on when we will have a web based FTP service back up and running. We were already running the latest version of net2ftp, so there isn’t a newer version we can upgrade to that will fix it. The software seems to be unmaintained now (the last update is over 2 years old at this point) so it’s not likely that we will be able to support it going forward. We’re currently looking at alternatives, but it is going to take some time to test and implement. As mentioned in the main post above, you will need to use an alternate FTP client for the time being. Sorry for the inconvenience.

Our security team has verified that this software was not used by an attacker to affect any of our systems. Any compromised site on your account that you have seen is unrelated and most likely due to outdated software. Please contact support to get in touch with our security team for a security scan.

Update – Oct 24th 10:00AM (PST): After hours of work and strenuous internal testing, we now have WebFTP back online at:

https://webftp.dreamhost.com

We’ve closed the security hole that led it to being disabled and it should function the same as it had before. We’d like to reiterate that the reason this took so long is because the net2ftp software that WebFTP runs on is no longer actively being developed, so we had to find our own solution to secure it. We apologize for the inconvenience this caused, but we have to maintain a secure hosting environment for our customers. If you have any problems with the WebFTP service now, please contact support and we’ll be happy to investigate it further for you.


4:08 pm

Web Panel sessions being reset. (RESOLVED)

Posted (October 4th, 2011 at 4:08 pm PST) by Oscar

At 4:30pm PST we will be resetting sessions for our Web Panel. This will affect all customers and will require you to log in again to the Web Panel. We sincerely apologize for the short notice, but this should not have any major effects other than requiring you to log in to the Web Panel again.


12:19 pm

Shared sever ‘peppy’ offline for unscheduled file-system check (RESOLVED)

Posted (October 1st, 2011 at 12:19 pm PST) by mir

The shared web hosting server ‘peppy’ has been taken offline for an unscheduled file-system check. The server rebooted to a read-only file-system, which is a sign of issues with the drives, so this check is being performed to verify the integrity of the data on the server. This is expected to take no more than 15 minutes. -John

Update Oct. 1, 12:30pm PST: This issue is now resolved, thank you for your patience. -John


8:39 am

Web Panel Outage (RESOLVED)

Posted (October 1st, 2011 at 8:39 am PST) by Oscar

Currently we are experiencing an outage with our web panel. Our admin team is working on the issue and we should have this resolved shortly. We will continue to update this post as soon as we have more information.

Update Oct. 1, 8:45am PST: The web panel is now back up and should be working without any issues. -Oscar


4:01 pm

Operating System Upgrade for 50 machines on Oct. 3rd 7pm PDT – machine names included (RESOLVED)

Posted (September 30th, 2011 at 4:01 pm PST) by Ralph

On Monday Oct. 3rd at 7pm PDT we’ll be upgrading the operating system to Debian “squeeze” for the following machines:

talitha-borealis tallinn tarawa-atoll tarazet tashkent taurus tbilisi tech21 tegucigalpa tehran telescopium terranova texans theia themis thimphu thing thor thrashers thunder thunderkittens tick tigercats tigers tirane titans tobias tokyo torr torshavn traceelliot trafficante triangulum tripoli tucana turmagar twins twoface tygra ulaanbaatar univox ursamajor ursaminor vaduz valletta valleyarts vaticancity vela venom

5 machines will be upgrade at a time starting a 7pm and those upgrades should last around 30-45 minutes. We will then continue with another 5 after the first 5 have completed successfully. We will be upgrading servers in the order listed above. Our test runs went relatively smoothly so we do not expect any issues, but we will be watching these upgrades closely just the same.

We apologize for the downtime these upgrades will cause and we will do our best to limit the downtime to just the time it takes to do the upgrade.

Update Oct 4th 11:15am PDT: Sorry about the lack of updates – this upgrade has been completed earlier today. Please contact Support if you are having any issues with your hosting following these upgrades! -Andrea

Previous updates available after the jump →


9:01 am

Distributed Denial of Service Attack causing networking issues on SEVERAL machines (RESOLVED)

Posted (September 30th, 2011 at 9:01 am PST) by mir

Our initial calculations were off. This outage is impacting more machines than we anticipated. If your machine is down at the moment, you are likely affected by this outage. Sorry that I could not provide a list of all machines.

Our admin team is working on the matter and hopes to have it resolved in the very near future. We’ll keep this post updated as information becomes available. Please note that the servers themselves are not compromised (this sort of attack utilizes exploits of outdated applications installed by users to launch an overwhelming number of data requests which bogs down the networking on the servers involved – the distributed nature of the attacks make them very difficult to fix as it’s not just a matter of blocking a single IP or IP range).

Update Sept. 30th, 10:05am PST: We have confirmed that this issue has been resolved, thank you for your patience. – John

Previous updates available after the jump →


6:05 am

Shared web server ‘reykjavik’ offline for unscheduled file-system check (RESOLVED))

Posted (September 30th, 2011 at 6:05 am PST) by patrickm

The shared web hosting server ‘reykjavik’ has been taken offline for an unscheduled file-system check.  The server rebooted to a read-only file-system, which is a sign of issues with the drives, so this check is being performed to verify the integrity of the data on the server.  We will update this post as this process continues and once we have a realistic ETA!

Update Sept. 30th, 5:35pm PST: Our admin team have given us the all clear. The file system check completed on the original machine and no backup machine was needed for this. Our sincerest apologies for the downtime and all sites are now back up. If you are having any issues with your sites being down, please contact our support staff and we will be glad to look into it for you. -Oscar Previous updates available after the jump →


 
© 1996-2011, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
9:44 am

VPS host homie-vserver197 down for unscheduled file system check (RESOLVED)

Posted (September 29th, 2011 at 9:44 am PST) by Justin@DH

Our datacenter admins noticed this vserver host machine go read-only a short time ago. After a reboot, it seemed ok again, but then it happened again. Unfortunately we will need to run a file system check on this server to correct issues that are causing this. We will update this post regularly with updates on how far along it is, but unfortunately an ETA is more difficult due to file system checks varying with number of files, size of the partition and how full it is.

The affected VPS servers are:

ps11126 ps11137 ps11277 ps11323 ps11594 ps11599 ps12547 ps14044 ps14047 ps14049 ps14050 ps14055 ps14107 ps14236 ps14471 ps14483 ps15212 ps15218 ps16414 ps16995 ps18375 ps18445 ps18447 ps19297 ps20194 ps22433 ps22758 ps24132 ps36902 ps36926 ps36957 ps36960 ps36972 ps36986 ps36990 ps38608 ps38681 ps38690 ps38697 ps38718 ps38724 ps38730 ps38733 ps38734 ps38738 ps38739 ps38743 ps38744 ps38746 ps38747 ps38748 ps38750 ps38752 ps38753 ps38754 ps38755 ps38756 ps38757 ps38760 ps38762 ps38764 ps38765 ps38766 ps38770 ps38772 ps38773 ps38774 ps38775 ps38776 ps38777 ps38778 ps38779 ps38780 ps38781 ps38782 ps38784 ps38787 ps38788 ps38789 ps38790 ps38791 ps38792 ps38795 ps38796 ps38797 ps38798 ps38799 ps38800 ps38802 ps38803 ps38804 ps38805 ps38806 ps38809 ps38812 ps38818 ps38819 ps38820 ps38821 ps38822 ps38829 ps38830 ps38832 ps38834 ps38838 ps38843 ps38846 ps38847 ps38850 ps38855 ps38858 ps38859 ps38862 ps38867 ps38868 ps38874 ps38878 ps38880 ps38881 ps38886 ps38889 ps38890 ps38891 ps39348 ps58381 ps58385 ps58463 ps58492 ps58498 ps59864 ps60848 ps61109 ps62589 ps62592 ps6821 ps6822 ps6825 ps6827 ps7006 ps7142 ps7176 ps7377 ps7421 ps7840 ps7864 ps8305 ps8710 ps8828 ps9038 ps9501 ps9544

Update Sept. 29th, 5:10pm PST: The file system check has finished and all guest VPS machines are now back up. Should you have any issues with your VPS still being down, please contact our support team and we will be glad to check on it for you. -Oscar

Previous updates available after the jump →


9:17 pm

Operating system upgrade for servers redwings and oilers on Sept 29th, 7pm PDT! (RESOLVED)

Posted (September 28th, 2011 at 9:17 pm PST) by Brian@dreamhost

We’ll be upgrading the operating system to Debian “squeeze” for servers redwings and oilers on Sept 29th at 7pm PDT. We expect each server to be down about 30 minutes. We apologize for the downtime this causes!

Update Sept. 29th, 6:32pm PST: Due to some other things that had to be tended to tonight, our admins have postponed this upgrade for the foreseeable future. As soon as we have another date and time, we’ll let you know! – Jason


11:46 am

MySQL Server Slowness Issues in Downtown LA Datacenter

Posted (September 28th, 2011 at 11:46 am PST) by Justin@DH

We are noticing speed and downtime issues with specific mysql servers in a specific vlan. Our network admins have determined that an uplink has become saturated, which is causing minor packet loss and slowness. There are plans to add an additional link which will fix this issue and we are working to get it done as soon as we can. In the meantime, unfortunately customers on the shared mysql servers:

anchovy bonito bullwinkle chapulin coelacanth croaker dopey dory dumbo gill glockenspiel grouper gumby gurgle halibut jetsons kazoo mackerel magoo narwahl piccolo pigwidgeon popeye puffer pumbaa salmon smelt spaceghost tambourine tetra topcat zazu

these VPS host machines:

homie-vsql31 homie-vsql32 homie-vsql33 homie-vsql34 homie-vsql35 homie-vsql36 homie-vsql38 homie-vsql41 homie-vsql78 homie-vsql79 homie-vsql80

May see slower than normal performance until this upgrade is performed. When we have more information or an ETA, we will update this post.

Update Sept. 28th, 12:03am PST: It appears our network admins were able to get this resolved faster than expected. Extra gigabit links have been added to the specific uplinks needed and performance seems much improved. If you are still seeing slowness and you are on the above servers, please contact support.


8:47 am

Easy One-Click Installs Downtime due to DDOS ( RESOLVED )

Posted (September 28th, 2011 at 8:47 am PST) by Justin@DH

The load balanced servers that are used for easy one-clicks are having issues at this time due to what appears to be a DDOS. We have several admins and a network admin on the case, looking at the cause and to get this fixed up as soon as we can. When we have further info on the cause and/or an ETA, we will update this post.

This issue only affects one-click installs in simple/easy mode, and does not affect advanced one-clicks, shared webservers, mail, mysql or any other service.

Update Sept. 28th, 12:04pm PST: This issues with the easy mode one-clicks are almost resolved. All of the machine are back online. The software installs are functional as well. You should be able to resume access to your easy mode software installs.

However, there is one caveat. We had to disable the registration feature on Drupal and PhpBB. Since this appeared to be the source of the exploit. Therefor new users cannot register on your sites. Our admins are still working on a more practical solution. Again we sincerely apologize for any inconvenience this has caused. Your continued patience is greatly appreciated. Please check back for updates.-Brian S

Update Sept. 28th, 11:27am PST: We have isolated the problem to PhpBB and Drupal installs. We have taken all ” easy mode ” phpbb and drupal installs offline for the moment. Also, one out of the four onceclick machines remain offline. We are diligently working towards bringing the machine back online. So you will continue to notice intermittent downtime while accessing other ” easy mode ” software installs as well ( WordPress, Zenphoto, etc )

Again, this outage only affects users who installed one-click software in ” easy mode ” through the Goodies::One-Click section of our Panel https://panel.dreamhost.com/index.cgi?tree=goodies.installer&. We sincerely apologize for any inconvenience this has caused and appreciate your continued patience. Please check back for future updates. -Brian S

Previous updates available after the jump →


5:41 pm

Operating system upgrade for lions, eagles, rams & bears on Sept 28th, 7pm PDT! (RESOLVED)

Posted (September 27th, 2011 at 5:41 pm PST) by Brian@dreamhost

We’ll be upgrading the operating system to Debian “squeeze” for servers lions, eagles, rams & bears on Sept 28th at 7pm PDT. We expect each server to be down about 30 minutes. We’ll do two @ 7pm and another 2 at 7:45. We apologize for the downtime this causes!

UPDATE: Sept 28th, 9:15pm PDT This update is complete. If you notice any problems with your account and you’re on these servers, please let us know ASAP! — Brian H


6:43 pm

Upgrading shared server ‘padres’ to Debian ‘squeeze’ at 7pm PST on 9/27 (RESOLVED)

Posted (September 26th, 2011 at 6:43 pm PST) by Jason C

Our admin team will be upgrading the operating system on the shared server ‘padres’ on Tuesday, Sept. 27th at 7pm PST. We estimate that the downtime for this upgrade should be no more than 30-60 minutes. If any additional details bubble to the surface about this upgrade, you can be sure that we’ll post them here.

To find out if you will be affected, check the machine your user is on in the panel.

Update Sept. 28th, 2:30am PST: Our admin team has let us know that the upgrade has been completed and the server is running without any visible issues. Should you happen to notice any problems with your sites on padres, however, please don’t hesitate to contact our support team. – JJ G


1:40 pm

Scheduled Maintenance for “homie-vserver190″ on 2011-09-26 at 11pm PDT (RESOLVED)

Posted (September 26th, 2011 at 1:40 pm PST) by Justin@DH

The webserver homie-vserver190 which hosts the following webserver VPS:

ps30158 ps30168 ps30170 ps30172 ps30175 ps30176 ps30177 ps30179 ps30180 ps30181 ps30184 ps30190 ps30191 ps30193 ps30194 ps30195 ps30197 ps30198 ps30201 ps30202 ps30203 ps30204 ps30205 ps30206 ps30207 ps30208 ps30212 ps33594 ps34244 ps35663 ps35742 ps35992 ps36226 ps36379 ps36880 ps37052 ps44503 ps44509 ps44521 ps44580 ps46681 ps48397 ps50567 ps50633 ps52338 ps53583 ps53585 ps55549

has been having repeat issues with its RAID, where it would go read-only. The admins wish to do a file system check on it, as preventative maintenance before more serious RAID issues occur. The file system check will require the VPS to be shut down, and will begin at 11PM pacific time. We will update the status post as it progresses to give a rough ETA and % done update. If you have any questions, please contact support.

Update Sept 27th, 2:12am PST: This file-system check is now complete and the server is back online! If your VPS is not responding, or you notice any other issues, please contact Support right away! -PatrickM

Previous updates available after the jump →


8:24 am

VPS Host Machine “homie-vserver25″ being moved to new hardware (RESOLVED)

Posted (September 26th, 2011 at 8:24 am PST) by Justin@DH

The homie-vserver25 server has had a history of RAID related issues and needing of file system checks. This whole time we have been copying its data to new hardware to replace it as soon as it completed, which is today. We will be taking the old homie-vserver25 down and replacing it with new hardware which has the current data copied to it. Our admins don’t expect more than 30 to 45 minutes to complete the process, re-configuration, etc, but we will update this post with the progress as it happens. Once this is complete, this vserver host machine should be back to stable and normal service.

Update Sept 26th, 9:25am PST:

My apologies for not including the list of VPS that use this host machine. The affected VPS customers are:

ps10037 ps10941 ps17558 ps17559 ps17561 ps17562 ps18199 ps18254 ps19572 ps19614 ps19668 ps26301 ps26503 ps27037 ps27044 ps30777 ps39340 ps39382 ps39384 ps39386 ps39586 ps39635 ps39866 ps39940 ps40075 ps46444 ps46448 ps47486 ps48158 ps48477 ps48514 ps49162 ps49526 ps50460 ps50555 ps52172 ps60825 ps6636 ps6641 ps6674 ps6789 ps6792 ps6793 ps6794 ps6972 ps6974 ps7159 ps7184 ps7357 ps7398 ps7466 ps7514 ps7731 ps7790 ps7826 ps7881

I just heard back from the admins that the new hardware is up and all but 3 VPS are up and running. The admins are still working on fixing up the last 3, which are ps7790, ps7826 and ps7881.

If you notice any issues with missing files, please contact support, and we will do our best to recover any data from the old hardware. -Justin

Update Sept 26th, 11:43am PST: The three remaining servers (ps7790, ps7826 and ps7881) are back up and running now. A bunch of OS files needed to be rebuilt on them, which delayed getting those back up. We’re currently in the process of rsyncing the data in the /home directories for those servers and this issue should be fully resolved when those are finished. We don’t have an ETA on when this will complete, but will update this status post as we have more information. -dan w

Update Sept 26th, 12:20pm PST: The rsyncs on these final three servers finished about 15 minutes ago. We’ve tested out the server as much as we can on this side and things seem to be working smoothly now. Hopefully on the new hardware this server will be a lot more stable. If you have any further issues regarding your VPS server on this host machine, please contact support for further assistance. -dan w


3:35 pm

VPS host homie-vserver102 down for unscheduled file system check

Posted (September 25th, 2011 at 3:35 pm PST) by Jason C

While our admins thought that we could manage to keep homie-vserver102 stable after its previous issues earlier today, it looks like that was not the case. As of right now, homie-vserver102 is back down for an unscheduled file system check.

Because of this rash of problems with the filesystem on the server, our admin team is working on failing the machine over to new hardware as well. There’s a possibility that we won’t need to use it, but if we do, we want to be ready.

Here’s a list of affected VPSes on that homie-vserver102:

ps16497 ps16868 ps16872 ps16879 ps16890 ps16893 ps16943 ps17088 ps17531 ps17533 ps17568 ps17576 ps17585 ps17588 ps17652 ps17924 ps18279 ps18308 ps23969 ps24033 ps24610 ps26784 ps30609 ps33820 ps35506 ps35520 ps35561 ps35604 ps36160 ps36784 ps37078 ps38914 ps39379 ps42734 ps50395 ps51515 ps53549 ps53728 ps54448 ps54577 ps54589 ps54591 ps55260 ps55561 ps56042 ps56981 ps61378 ps61660

Keep an eye here for updates as this matter progresses.

Update Sept. 25th, 11:42pm PST: The server has completed its file system check and has been up and running without issues for about 15 minutes. If you have any issues with any of the VPSes listed above, please contact support!

Previous updates available after the jump →


12:46 am

Host machine homie-vserver102 downtime for unplanned file-system check (RESOLVED)

Posted (September 25th, 2011 at 12:46 am PST) by jj

The host machine homie-vserver102 is having issues with the file system and is having a file system check executed. We will update this post as the file system check progresses. The affected VPS machines are:

ps16497 ps16868 ps16872 ps16879 ps16890 ps16893 ps16943 ps17088 ps17531 ps17533 ps17568 ps17576 ps17585 ps17588 ps17652 ps17924 ps18279 ps18308 ps23969 ps24033 ps24610 ps26784 ps30609 ps33820 ps35506 ps35520 ps35561 ps35604 ps36160 ps36784 ps37078 ps38914 ps39379 ps42734 ps50395 ps51515 ps53549 ps53728 ps54448 ps54577 ps54589 ps54591 ps55260 ps55561 ps56042 ps56981 ps61378 ps61660

Update: Sept 25, 3:35am: Everything seems to be up and running properly. The VPS machines on this host are running smoothly, if you have any problems with a server on the list above please let support know. -JJ G Previous updates available after the jump →


 
© 1996-2011, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
8:14 pm

Upgrading shared server ‘lakers’ to Debian ‘squeeze’ at 7pm PST on 9/26 (RESOLVED)

Posted (September 23rd, 2011 at 8:14 pm PST) by Brian@dreamhost

We’re preparing to upgrade the operating system on the shared machine ‘lakers’ on Monday, Sept 26th at 7pm PDT. We estimate the downtime for this upgrade to last no more 30-60 minutes. We’ll update this post with any additional details we receive about this upgrade.

To find out if you will be affected, check the machine your user is on in the panel.

Update Sept. 26th, 7:47pm PST: Our admin team has let us know that the upgrade has been completed and the server is running without any visible issues. Should you happen to notice any problems with your sites on lakers, however, please don’t hesitate to contact our support team. – Jason


4:53 pm

Homie-vserver163 down due to raid card failure (RESOLVED)

Posted (September 22nd, 2011 at 4:53 pm PST) by Oscar

The homie-vserver163 VPS host machine is unfortunately having issues again. The raid card has completely failed and our admin team is currently on route to the data center to replace the raid card. The progress to replace the raid card should take no longer than 15-20 minutes.  We will update this post as soon as our admin team starts the process . The affected VPS customers are:

ps23521 ps23593 ps23671 ps23710 ps23742 ps23781 ps23831 ps24069 ps24124 ps24166 ps24377 ps24537 ps24822 ps25005 ps25140 ps25169 ps25440 ps25595 ps25663 ps25721 ps25801 ps26099 ps27272 ps27629 ps27635 ps27900 ps27969 ps27987 ps28028 ps28370 ps29012 ps29207 ps34378 ps35860 ps36769 ps36809 ps36812 ps36824 ps36855 ps36944 ps37152 ps39236 ps43161 ps43403 ps43768 ps43942 ps44162 ps45048 ps46486 ps47085 ps47091 ps58282 ps58886 ps58892 ps58893 ps59367 ps59400

Update: Sept 22, 6:20pm: The raid card was replaced and the server is now back up. All VPS servers on this machine have been restarted and your sites are now back up. Should you have any issues with your sites being down, please contact our support team and we will be glad to look into it for you. -Oscar

Update: Sept 22, 5:38pm: Our admin team has arrived at the datacenter and the raid card replacement should take no longer than 10-15 minutes. We will update this post as soon as the machine is back online. -Oscar


1:35 pm

Network Outage at our LAX Datacenter (RESOLVED)

Posted (September 22nd, 2011 at 1:35 pm PST) by Justin@DH

We are currently looking at 3 specific racks of servers that stopped responding a short time ago. Our network engineer and admins are looking into the issue to get this resolved as soon as possible. We are also working on a list of affected servers. Specific ones we are currently aware of are vamika, flint and hambone, 3 shared mysql servers. More information shortly.

Update Sept 22th, 1:55pm PST: The service on these server racks have been restored. We’re not quite sure what the issue was, but we’re continuing to keep an eye on things to make sure they continue to run smoothly. We’ve tested the affected servers and everything is back to normal now. Sorry for any inconvenience.


10:10 am

Vserver host homie-vserver163 downtime for unplanned file-system check (RESOLVED)

Posted (September 22nd, 2011 at 10:10 am PST) by Justin@DH

The homie-vserver163 VPS host machine is unfortunately having issues as the file system is in an inconsistent state and needs to have a file system check. Our oncall admin is heading to the datacenter now to begin this process. We will update this post as the progress for the status post continues. The affected VPS customers are:

ps23521 ps23593 ps23671 ps23710 ps23742 ps23781 ps23831 ps24069 ps24124 ps24166 ps24377 ps24537 ps24822 ps25005 ps25140 ps25169 ps25440 ps25595 ps25663 ps25721 ps25801 ps26099 ps27272 ps27629 ps27635 ps27900 ps27969 ps27987 ps28028 ps28370 ps29012 ps29207 ps34378 ps35860 ps36769 ps36809 ps36812 ps36824 ps36855 ps36944 ps37152 ps39236 ps43161 ps43403 ps43768 ps43942 ps44162 ps45048 ps46486 ps47085 ps47091 ps58282 ps58886 ps58892 ps58893 ps59367 ps59400

Update Sept 22th, 2:40pm PST: The server is back up and running. We’ve tested several VPS servers on this machine and everything looks like it is back to normal now. If you’re on one of the VPS servers above and are still having issues, please contact support for further assistance.

Previous updates available after the jump →


4:36 pm

Homie-vserver190 down due to raid card failure (RESOLVED)

Posted (September 21st, 2011 at 4:36 pm PST) by Oscar

Homie-vserver190 is currently having issues with it’s raid card. Our admin team will be replacing the raid card shortly as they are on route to the data center. We will be keeping the host machine down until the raid card has been replaced in order to avoid filesystem corruption. The raid card replacement should take no longer than 10-15 minutes once the process has started. We will update this post as soon as we have more information. –Oscar

Affected machines are:

ps30158 ps30168 ps30170 ps30172 ps30175 ps30176 ps30177 ps30179 ps30180 ps30181 ps30184
ps30190 ps30191 ps30193 ps30194 ps30195 ps30197 ps30198 ps30201 ps30202 ps30203 ps30204
ps30205 ps30206 ps30207 ps30208 ps30212 ps33594 ps34244 ps35663 ps35742 ps35992 ps36226
ps36379 ps36880 ps37052 ps44503 ps44509 ps44521 ps44580 ps46681 ps48397 ps50567 ps50633
ps52338 ps53583 ps53585 ps55549

Update Sept 21th, 6:30pm PST: The server is now back up and all VPS servers have been restarted again. Should you have any issues with your VPS or sites being down, please contact our support team and we will be glad to look into the issue for you. — Oscar Previous updates available after the jump →


3:12 pm

CafeCommerce database issues (RESOLVED)

Posted (September 21st, 2011 at 3:12 pm PST) by andrea

If you’re running a CafeCommerce store, you may be experiencing intermittent downtime. We’re looking into the problem and would like to apologize for any inconvenience this may be causing. We’ll update this post when we know more. – Andrea

Update Sept 21, 3:37pm PST:  This is fixed now and all sites should be working without any issues. Our admin team was able to correct the issue which was causing mysql to stop and restart. If you have any further issues with your Cafe Commerce sites now, please contact our support staff and we will be glad to look into for you. Oscar

 


1:12 pm

Upgrading shared server ‘packers’ to Debian ‘squeeze’ at 7pm PST on 9/22 (RESOLVED)

Posted (September 21st, 2011 at 1:12 pm PST) by andrea

We’re preparing to upgrade the operating system on the shared machine ‘packers’ tonight at 7pm. We estimate the downtime for this upgrade to last no more 30-60 minutes. We’ll update this post with any additional details we receive about this upgrade.

To find out if you will be affected, check the machine your user is on in the panel. -Andrea

Update: Sept 22, 7:40pm: The upgrade is all done and the server is back up. Should you have any issues with your sites being down, please contact our support staff and we will be glad to look into the issue for you. -Oscar Previous updates available after the jump →


2:13 pm

Shared server “bishop” down due to hardware issues (RESOLVED)

Posted (September 20th, 2011 at 2:13 pm PST) by Oscar

The shared server “bishop” is currently down due to an issue with it’s boot disk drive. The primary boot disk lost its root filesystem and we are currently re-imaging it now. This should take about 1-2 hours. We will update this post as soon as we have more information. –Oscar

Update Sept 20th, 5:15pm PST:  All apache instances have been reconfigured and restarted. All of your sites are now resolving without any issues. If you do notice any issues with your site or it’s down, please contact our support team and we will be glad to look into it for you. -Oscar

Previous updates available after the jump →


7:44 am

Issues with new process watcher causing downtime (RESOLVED)

Posted (September 20th, 2011 at 7:44 am PST) by mir

I apologize for the downtime you are experiencing this morning, we have a number of apaches services down currently due to issues with a new version of our process watcher. We’re currently working on this and should have it resolved shortly (it just took a little while to determine the cause as the various servers didn’t initially seem to have anything in common causing the outages). We’ll update this post regularly with details until the matter is resolved. -John

Addendum:

What happened was that when we rolled out our improved process watcher service early this morning it wasn’t properly started on all machines. The result was what would happen if we didn’t run the process watcher service at all (crashed apache web servers and hosting machines all over the place due to out of control user processes) – it’s a good thing that this part of our system is usually there and working properly! Once we fixed that and started it up we just had to correct individual servers which had other issues. All that has been handled now and we have closed out support related to this issue so if you are still having trouble please contact support with details and we’ll take care of it for you.

Update Sept 20th, 10:05am PST: The admin team has assured me that the apache web servers are all back up now so this matter is now resolved. If your issue has still not been corrected it just need individual attention so please let us know and we’ll get right on it! -John

Previous updates available after the jump →


6:26 am

Shared server “angel” down for file-system check (RESOLVED)

Posted (September 20th, 2011 at 6:26 am PST) by mir

The shared server “angel” had to be rebooted and when it came up a fsck started. This was not scheduled but unfortunately is not something we’re able to circumvent so we have to let it run its course. We’ll update this post regularly to keep you informed. -John

Update Sept 20th, 5:00pm PST: All apache instances are back up now and your sites are now resolving. Should you have any further issues with your sites being down, please contact our support team and we will be glad to look into it for you. -Oscar

Previous updates available after the jump →


 
© 1996-2011, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
3:40 pm

Shared server ‘luxembourg’ offline due to RAID card failure. (RESOLVED)

Posted (September 19th, 2011 at 3:40 pm PST) by Brandon

The shared webserver ‘luxembourg’ is currently offline while we replace it’s RAID card. The machine should be back up shortly, and we will update this post with new information when it becomes available.

 

Update Sept 19th, 3:53pm PST: The RAID card has been successfully replaced and all customer sites should be back online. Please contact our Technical Support department should you still be seeing any issues.


2:30 pm

Intermittent connection issues on ‘homiemail-sub4′ mail cluster (RESOLVED)

Posted (September 19th, 2011 at 2:30 pm PST) by Brandon

One of the servers in the ‘homiemail-sub4′ mail cluster has been removed from the cluster, resulting in higher than normal load averages on the remaining machines. We are currently working on replacing it with two new machines, which should alleviate the problem. In the meantime, some customers may be intermittently receiving IMAP connection errors. Any mail being sent should be sent in a timely matter, without delays.

We’ll update this post with new information as it becomes available.

Update Sept 20th, 12:20am PST: This issue was resolved as of yesterday afternoon, so our apologies for leaving this post without an update for so long. This server was placed back in the cluster and has been performing well all day!  We are not seeing any new reports of trouble, but please contact Support if you have any questions, or issues! -PatrickM


1:49 pm

Upgrading shared server ‘univox’ to Debian ‘squeeze’ at 7pm PST on 9/20 (RESOLVED)

Posted (September 19th, 2011 at 1:49 pm PST) by andrea

We’re preparing to upgrade the operating system on the shared machine ‘univox’ tomorrow night, at 7pm. We estimate the downtime for this upgrade to last no more 30-60 minutes. We’ll update this post with any additional details we receive about this upgrade.

To find out if you will be affected, check the machine your user is on in the panel. – Andrea

 

Update Sept 20th, 10:21pm PST:The upgrade was completed approximately 40 minutes ago. If you notice any issues, please contact our support staff.


9:37 am

Shared server ” bern ” offline for a RAID replacement ( RESOLVED )

Posted (September 16th, 2011 at 9:37 am PST) by brians

The shared server ‘bern’ crashed due to a RAID card failure. One of our admins is currently on their way to the data center to replace that card. Once we start the process to replace the card, it should take no longer than 15-30 minutes to get the server back up. Please accept our apologies. Your patience is greatly appreciated. We will update this status post as soon as we have any additional information.

UPDATE Sept 16, 10:36AM PST: The machine is back online. All services are fully functional. Again, we sincerely apologize for any inconvenience this has caused. If you are still having problems, please contact support immediately. We will gladly take a look at your case.


7:25 am

Sending of Mail Messages from Sites and Scripts Delayed Temporarily (RESOLVED)

Posted (September 16th, 2011 at 7:25 am PST) by mir

A couple of our outgoing mail servers have exceedingly large queues which is resulting in some delays in sending messages through those machines. This will not prevent mail from being delivered but in cases where mail is routed through those servers mail delivery will be delayed. This will clear itself up organically as the messages are processed but our administration team is working on speeding up the process as well. We’ll update this post once we have more information for you. -John

UPDATE Sept 16th, 8:35PM PST: Ok, this should be completely clear now, mail should be delivered normally. If you’re still seeing problems, please let support know asap! — Brian H

Previous updates available after the jump →


9:55 pm

Scheduled maintenance for homie-vsql89 on Sunday, September 18th @ 9pm PDT (RESOLVED)

Posted (September 15th, 2011 at 9:55 pm PST) by Brian@dreamhost

The mysql VPS host server “homie-vsql89″ needs to be taken down for maintenance on Sunday, Sept 18th at 9pm PDT. We expect this will take about 20-30 minutes, possibly less. The following servers will be affected:

psmysql58358 psmysql58375 psmysql58382 psmysql58422 psmysql58437 psmysql58438 psmysql58447 psmysql58448 psmysql58455 psmysql58456 psmysql58473 psmysql58481 psmysql58523 psmysql58535 psmysql58552 psmysql58553 psmysql58887 psmysql58940 psmysql58959 psmysql58974 psmysql59083 psmysql59089 psmysql60246 psmysql60353 psmysql60362 psmysql60761 psmysql60796 psmysql60797

Sorry about the upcoming downtime on these servers!

Update Sept 18th, 9:32pm PST: The maintenance on this machine is now complete. Thank you for your patience. If you’re experiencing any issues with your sites related to this mysql VPS host, please contact support with the details.

Please note, this was only affecting the machines listed above. As always, if you have any sort of other issues, a detailed ticket for support is your best bet to get help.


1:30 pm

Scheduled internal network maintenance and hardware upgrade on the night of 9/21 and 9/22 (RESOLVED)

Posted (September 15th, 2011 at 1:30 pm PST) by andrea

We’ll be conducting network maintenance and hardware upgrade on the night of 9/21 and 9/22, between 8pm and midnight pacific time. This is internal maintenance, and as such it should not affect customer sites. However, there may be intermittent downtime, up to 30 minutes each, so please plan accordingly.

Please contact support with the details if you experience any issues during this time or to find out ahead of time if and which day you may be affected. – Andrea

Update Sept 22nd, 9:15pm PST: The 2nd half of this upgrade to our network is postponed for the time being. Our hardware provider is having trouble confirming a few things, so we are not going to update the network at this time. We’ll post an update date when we are able to give a date when we’ll finish the rest of this. — Brian h

Update Sept 22nd, 3:50am PST: The issues caused by this maintenance (9/21 only) are now completely resolved! To summarize, we had a problem with the fiber connection for one row of servers, which caused the initial downtime. Next we took down a border router and this caused some unexpected connection issues. Finally, the bad fiber in that row completely quit on us and it was replaced. As of 3:00am PST, all downtime from this should have been resolved.

If you are experiencing any issue with your hosting at this time, please contact Support so we can be sure you are all set! -PatrickM

Update Sept 22nd, 3:25am PST: We are reluctant to resolve this post, as we had another issue with some bad fiber for one row of servers. We are working through all the tickets that have been submitted on this to ensure this is totally resolved before calling it fixed. We should know more shortly and will update again as soon as possible! -PatrickM

Update Sept 22nd, 2:20am PST: We apologize for the downtime caused by this maintenance. We are still experiencing connectivity issues with some ISPs and are working hard to resolve the issues at this time. We will update again shortly! -PatrickM

Update Sept 22nd, 12:50am PST: This maintenance window is being extended to 2am PST.  We have resolved the issues causing downtime, but now need more time to complete the actual work.  We will update this post shortly with more info! -PatrickM

Update Sept 22nd, 12:10am PST: We apologize as this issue is still ongoing and is now affecting some MySQL connections. Our network team is hard at work resolving this issue and we will post when we have more details! -PatrickM

Update: Sept 21, 10:15pm PST: The downtime actually occurred on only one of the three mentioned rows (R), and the issue has been resolved. All sites should be back up but we will continue to post updates as maintenance continues. — JJ G

Update: Sept 21, 10:00pm PST: The scheduled maintenance is currently causing downtime for customers in the LAX data center, rows R, Q, and E. We’re working on this and will update this post when the issue is resolved or more information is available. — JJ G


10:15 am

Shared server funafuti downtime for RAID issues (RESOLVED)

Posted (September 15th, 2011 at 10:15 am PST) by Justin@DH

The shared webserver funafuti is having stability issues due to a hardware issue, in this case the raid card. We have an admin on location already and the situation is being looked at. Once we have further info about the case, we will update this post.

Update Sept 15th, 10:23am PST: The raid card has been replaced and the server is back up. If you notice further issues, please contact support. -Justin


7:07 am

Shared server “cardiff” downtime for unplanned file-system check (RESOLVED)

Posted (September 15th, 2011 at 7:07 am PST) by Justin@DH

The cardiff shared webserver had its operating system partition go read-only due to file system errors, and our admins have had to take the server down to run a file system check to get it working normally again. So far only the operating system partition is showing any signs of issues, and we are hoping a file system check on that data partition won’t be necessary also, since that takes much much longer. We will update this post soon with the status of the file system check or if the all-clear can be given.

Update Sept 15th, 8:00am PST: The file system check finished, the raid card was replaced and the server is back up and seems to be working normally. We will keep an eye on it for further issues but for now this issue appears resolved. -Justin


2:36 am

Shared server tucana moving to new hardware ( RESOLVED )

Posted (September 15th, 2011 at 2:36 am PST) by patrickm

The shared web hosting server ‘tucana’ is coming offline for a total hardware swap.  This will involve a live-restoration of data from backups, which will allow all hosted sites to remain online while this process completes through an NFS mount to the backup server.  The restore should take a number of days to complete and we will be closely monitoring the server for issues with high load arising from the data transfer.  We will be updating this post throughout the process, so check back here or contact Support if you have questions!

Update Sept 15th, 3:20pm PST: We have great news!!! One of our crafty admins was able to save the old machine. Meaning, a hardware replacement was not necessary. This whole outage was just a whole bunch of nothing. We are migrating back to the original as I type. There will be a brief period of downtime. Since we are going to upgrade the raid card.

Please note that any changes you made to your data between this morning and NOW are not live. However, they are retrievable. Just submit a support request and we will gladly lend a hand. Please provide the full path ( ex. /home/user/foldername/filename ) and we will handle the rest.

Again, we sincerely apologize for any inconvenience / panic that this false alarm caused. Your continued patience is greatly appreciated. – Brian S
Previous updates available after the jump →


 
© 1996-2011, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
2:04 am

Shared server tucana downtime for RAID issues (RESOLVED)

Posted (September 14th, 2011 at 2:04 am PST) by patrickm

We regret to report that the shared web hosting server ‘tucana’ is offline at this time due to an issue with the RAID controller.  We are working on restoring service to this server at this time and will update this post shortly!  Please contact Support with any questions you have!

Update Sept 13th, 8:10am PST: All apache services are running and we believe the server is back to normal now. If you notice further issues, please contact support with details. – Justin

Previous updates available after the jump →


5:25 pm

Scheduled network maintenance Thursday night – not customer facing – POSTPONED

Posted (September 13th, 2011 at 5:25 pm PST) by andrea

We’ll be conducting network maintenance on Thursday (9/15) night, between 10pm and midnight pacific time. This is internal maintenance, should not affect customer sites, but please contact support with the details if you experience any issues during this time. – Andrea

Update Sept 15th, 1:27pm PST: Just received word that this maintenance is postponed. Closing this status and will post a new one with the new details. – Andrea


6:12 pm

Upgrading operating system of shared server “conakry” to squeeze on Sept. 13, 2011 (RESOLVED)

Posted (September 12th, 2011 at 6:12 pm PST) by Jason C

Good news everyone! We’re ready to start rolling out the latest version of Debian (squeeze) to our shared server customers starting tomorrow.

In order to make sure that the process is as seamless as possible, we’re starting with a single server (conakry) at 7:30pm PST.

While this upgrade takes place, we will update this post with any and all changes in the status of the server. Hopefully, you won’t even notice a thing.

If you have any concerns leading up to the upgrade, please don’t hesitate to contact our support team. They’ll be more than happy to answer any questions you might have!

Update Sept 13th, 10:53pm PST: This is now complete!  We hit a snag or two with the process, but have ironed out the kinks and will be able to better upgrade future servers.  If you are experiencing any issues with your hosting now that the server is back online, please contact Support! -PatrickM

Previous updates available after the jump →


3:49 pm

The ‘wolfsbane’ server having possible kernel issues (RESOLVED)

Posted (September 12th, 2011 at 3:49 pm PST) by danwy

The wolfsbane server is having some issues with its kernel after rebooting it due to stability issues. We don’t have much information at this time, but our admins are investigating it further. If your site on this server is slow or non-responsive, we are looking into the cause and hope to have it back to normal soon. We’ll update this status post shortly with more information.

Update Sept 12th, 4:39pm PST: Our admin team spotted an issue with a network card while poking around a bit more, so a few extra reboots were needed to straighten it all out. Things continue to stay stable after the reboot, but we may need to replace the card at a later date. If we do, you can be sure that there’ll be another post here detailing that.

Previous updates available after the jump →


2:01 pm

Shared web server ‘gravano’ downtime (RESOLVED)

Posted (September 12th, 2011 at 2:01 pm PST) by andrea

We’re currently experiencing some recurring downtime and slowness on the shared web server gravano. Our techs are looking into the cause of the issue. We’ll report back when we know more. Sorry about the inconvenience this is causing.

To find out if you’re on gravano, please go to your panel to check your ftp/shell user’s home machine. – Andrea

Update Sept 12th, 2:25pm PST: The server has been back up for the last 15 minutes and seems to be doing OK. We’re going to continue to keep an eye on the server to make sure things continue to run smoothly though. We’ll keep this status post open for a little while longer until we’re confident there will be no further issues. We’ll update this post again shortly if any new issues arise or if we decide to mark it as resolved. -dan w

Update Sept 12th, 3:00pm PST: The server has continued to perform with no issues, so we’re going to go ahead and resolve this post. We have multiple people keeping an eye on it though, so if any further issues do come up we will be able to deal with it quickly. We will also update this status post if that happens. Please contact support if you have any further issues with your sites on this server. -dan w


9:39 am

Incoming DDOS on Shared Machine ” Sceptrum ” ( RESOLVED )

Posted (September 12th, 2011 at 9:39 am PST) by brians

This would explain any type of network instability that you are encountering. All connections ( HTTP, SFTP, SSH, FTP, etc ) to the machine may result in intermittent failure. Our network engineers are on the case. We sincerely apologize for any inconvenience this has caused. Please check back for updates.

UPDATE Sept 12th, 9:50AM PST: We have pinpointed the destination target and renumbered the IP. Everything has been resolved. All services are functioning at optimal performance. We sincerely apologize for any inconvenience this has caused. If you are still having problems, please submit a support request. A member of our team will gladly assist you! – Brian S


10:58 am

Shared server “lilongwe” down for RAID replacement (RESOLVED)

Posted (September 11th, 2011 at 10:58 am PST) by Justin@DH

The shared server ‘lilongwe’ has been having issues lately related to raid card or disk issues. After the last time it went unresponsive, it did not return from a reboot and our admin is debugging the server issue now. The raid card and/or disks that are causing the issue will be replaced, and be believe at this time it can be repaired and not need to be restored from backups to new hardware. When we have further info on this server we will update this post.

Update Sept 11th, 4:07pm PST: The last reboot was done due to a firmware update, that initiated an auto reboot; no more problems :) We’re resolving this post, but please contact support if you need help with your domains; it’s likely unrelated to this outage. – Andrea

Previous updates available after the jump →


9:43 am

Vserver host homie-vserver25 downtime for unplanned file-system check (RESOLVED)

Posted (September 11th, 2011 at 9:43 am PST) by Justin@DH

The homie-vserver25 VPS host machine is unfortunately having further issues as the file system is again in an inconsistent state and needs a file system check. Our oncall admin is making a new machine in the background to restore from backups onto, so that we may move to new hardware if the issues continue. For now the file system check will continue until it completes or it is necessary to move to the new hardware. We will update this post as the progress for the status post continues.

The affected VPS customers are:

ps10037 ps10941 ps17558 ps17559 ps17561 ps17562 ps18199 ps18254 ps19572 ps19614 ps19668 ps26301 ps26503 ps27037 ps27044 ps30777 ps39340 ps39382 ps39384 ps39386 ps39586 ps39635 ps39866 ps39940 ps40075 ps46444 ps46448 ps47486 ps48158 ps48477 ps48514 ps49162 ps49526 ps50460 ps50555 ps52172 ps60825 ps6636 ps6641 ps6674 ps6789 ps6792 ps6793 ps6794 ps6972 ps6974 ps7159 ps7184 ps7357 ps7398 ps7466 ps7514 ps7731 ps7790 ps7826 ps7881

Update Sept 11th, 5:00pm PST: The file system check has finished on the original machine and all guests should be back up as of now. Just to be safe, our admin team is going to continue creating a brand new clone of homie-vserver25 from last night’s backups.

The reason we are using last night’s backups is to merely seed the process. If, at any point, we need to fail over to this new server, we will use our current backups to bring the new server up to date and minimize the amount of files that may be lost.

If you happen to notice any issues with your server, please contact our support team immediately. – Jason

Previous updates available after the jump →


8:06 pm

Homie-vserver25 having problems with boot file system (RESOLVED)

Posted (September 10th, 2011 at 8:06 pm PST) by Brian@dreamhost

Unfortunately, our server, homie-vserver25 is having issues, and needs a file system check. The machine will be down while this happens. The following servers will be affected:

ps10037 ps10941 ps17558 ps17559 ps17561 ps17562 ps18199 ps18254 ps19572 ps19614 ps19668 ps26301 ps26503 ps27037 ps27044 ps30777 ps39340 ps39382 ps39384 ps39386 ps39586 ps39635 ps39866 ps39940 ps40075 ps46444 ps46448 ps47486 ps48158 ps48477 ps48514 ps49162 ps49526 ps50460 ps50555 ps52172 ps6636 ps6641 ps6674 ps6789 ps6792 ps6793 ps6794 ps6972 ps6974 ps7159 ps7184 ps7357 ps7398 ps7466 ps7514 ps7731 ps7790 ps7826 ps7881

We’ll update this as we get more info!

Update 10:35pm PDT: Our admins were able to replace the raid controller on the original machine and report that the server is now up and running and all data should be intact. The server has been running smoothly for the last half hour as are the VPS services. If you are having any problems with a VPS on the list above please let support know.  — JJ G Previous updates available after the jump →


5:19 pm

Homie-vserver181 read-only, now needs an file system check (RESOLVED)

Posted (September 10th, 2011 at 5:19 pm PST) by Brian@dreamhost

Unfortunately, our server, homie-vserver181 is having issues, and needs a file system check. The machine will be down while this happens. The following servers will be affected:

ps27047 ps37465 ps37468 ps37521 ps38085 ps38277 ps38332 ps38356 ps38407 ps38410 ps38437 ps38439 ps38444 ps38447 ps38457 ps38475 ps38518 ps39013 ps39029 ps39031 ps39083 ps39084 ps39126 ps39445 ps39535 ps39541 ps39887 ps39941 ps40398 ps40662 ps40901 ps41442 ps41944 ps42325 ps45807 ps46862 ps46865 ps47711 ps47718 ps50428 ps50624 ps50658 ps50661 ps50665 ps50679 ps51834 ps52435 ps52497 ps53565 ps53733 ps55336 ps55400 ps56774 ps57214 ps57989

We’ll update this as we get more info!

Update Sept 11th 2:20am PDT: The check has completed and services have been restored to the servers affected. If you are still having any trouble with a machine listed above please contact support. — JJ G Previous updates available after the jump →


 
© 1996-2011, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
4:08 pm

Shared server “ophiuchus” down for RAID replacement (RESOLVED)

Posted (September 10th, 2011 at 4:08 pm PST) by mir

The shared server ‘ophiuchus’ crashed due to a RAID card failure. One of our admins is currently on their way to the data center to replace that card. Once we start the process to replace the card, it should take no longer than 15-30 minutes to get the server back up. We will update this status post as soon as we have any additional information. -John

Update Sept 10th 6:10PM PDT: This server should now be up and running. If you’re still seeing problems, contact support immediately. — Brian H

Previous updates available after the jump →


9:15 am

Secure Certificate (SSL) orders temporarily suspended (RESOLVED)

Posted (September 10th, 2011 at 9:15 am PST) by mir

We’ve discovered an issue with the API we use to order certificates from our provider. Unfortunately the error code being returned is not allowing an immediate fix so we’ve had to suspend the ordering of certificates until we can fix it. We realize that this is an important issue for our customers and our Development Team is focusing on getting this resolved as quickly as possible. We’ll update this post as soon as we have more information (or a fix in place). Thank you for your patience. -John

Update Sept 10th, 2:35pm PST: We have been testing this for the last few hours and everything has been working so this issue is resolved and certificates are once more available. Thank you for your patience! -John


4:22 pm

Shared server “ben-gali” down for RAID replacement (RESOLVED)

Posted (September 9th, 2011 at 4:22 pm PST) by Oscar

The shared server ‘ben-gali’ crashed due to a RAID card failure. One of our admins is currently on their way to the data center to replace that card. Once we start the process to replace the card, it should take no longer than 15-30 minutes to get the server back up.  We will update this status post as soon as we have any additional information.

Update Sept 9th, 5:55pm PST: The raid card for the server has been replaced and it has been restarted. At this time all sites should now be working. If you have any issues with your sites being down, please contact our support staff and we will be glad to look into it. –Oscar.


3:02 pm

Shared MySQL server ‘mooneye’ downtime (RESOLVED)

Posted (September 9th, 2011 at 3:02 pm PST) by Oscar

We are currently having issues with the shared mysql server mooneye.  Our admin team is already working on the issue and we will update this status post as soon as we have more information. Our sincere apologies for the downtime related to this issue.

Update Sept 9th, 3:20pm PST: Currently the server is checking and repairing tables. While this does take some time to complete, we need to let it finish to avoid any issues with data corruption. We will update this post once this process completes. –Oscar.

Update Sept 9th, 6:10pm PST: The table checking and repair process has completed and the mysql server is up now and working without any issues. Should you have any further issues with your sites not working, please contact our support staff and we will be glad to investigate the issue for you. –Oscar


12:29 pm

Homie-vserver188 will be taken offline for a RAID card replacement (RESOLVED)

Posted (September 9th, 2011 at 12:29 pm PST) by brians

The machine has a faulty raid card. This would explain why guests on this host are not functioning at optimal performance. I’ve pasted a list of the affected VPS guests below.

An admin is heading down to our datacenter facility to replace the raid card. In the meantime, you will continue to encounter periods of slowness, downtime, etc. This impacts all services ( http, sftp, ftp, ssh, etc ). All services should be restored within the next hour or so. Please bear with us until then. We sincerely apologize for any inconvenience this has caused and appreciate your continued patience. Please check back for future update.

ps28937 ps28940 ps28941 ps28942 ps28943 ps29425 ps29426 ps29427 ps29428 ps29429 ps29430 ps29431 ps29433 ps29467 ps29477 ps29481 ps29483 ps29484 ps29485 ps29529 ps37385 ps37392 ps37421 ps37461 ps39128 ps39177 ps39193 ps39202 ps39205 ps39231 ps39365 ps39544 ps39551 ps39877 ps39880 ps40637 ps40761 ps40808 ps41434 ps42011 ps42173 ps43185 ps44206 ps44316 ps44403 ps44825 ps44879 ps45131 ps45138 ps45728 ps47218 ps49946 ps50413 ps50486 ps53280 ps53466 ps54310 ps54325 ps54895 ps57645

Update Sept 9th, 1:55pm PST: This is happening now. We have shut down the machine and expect this to take no longer than 15 minutes. We will update this post as soon as the raid card replacement is done. –Oscar

Update Sept 9th, 2:00pm PST: This is now complete. The host machine is back up and all VPS machines on the host have restarted.  –Oscar


3:01 am

Vserver host homie-vserver201 downtime for unplanned file-system check (RESOLVED)

Posted (September 9th, 2011 at 3:01 am PST) by patrickm

We are very sorry to report that after this host server rebooting to a read-only file-system too many times, we are bringing it offline for a full file-system check.  We expect this process to take several hours and will update this post as the process continues.

The following VPS guests are affected by this outage:

ps40182 ps40300 ps40635 ps40676 ps40766 ps41033 ps41041 ps41154 ps41268 ps41296 ps41795 ps41867 ps41936 ps42026 ps42090 ps42097 ps42151 ps42171 ps42400 ps42708 ps42976 ps43680 ps43989 ps44053 ps44060 ps44069 ps44173 ps44194 ps44257 ps44429 ps44456 ps44656 ps44937 ps45445 ps45548 ps45944 ps46580 ps47021 ps47144 ps47179 ps47399 ps47532 ps47637 ps47767 ps47877 ps48049 ps50335 ps50605 ps51303 ps52502 ps52535 ps54151 ps54467 ps54468 ps55289 ps55522

Check back at this post for updates!

Please be aware that there are 5 passes in a file system check. We make note of them as 1a, 1b, 1c, 1d and then finally 2. For the most part 1a is the longest part of a file system check. We are working on ways to have your sites and files available while a VPS host is down for a file system check, but unfortunately we do not have that capability yet. We will continue to monitor the file system check and get this working for you as soon as possible!

Update Sept 9th, 8:55am PST: The file system check is completed and VPS guests are back up and running so this matter is resolved. -John

 

Previous updates available after the jump →


10:56 pm

Brief downtime for 16 Shared Web Hosting servers starting at 11:00pm PST (RESOLVED)

Posted (September 8th, 2011 at 10:56 pm PST) by patrickm

We need to quickly re-assign the power ports for the following shared servers:

windhoek, kathmandu, amsterdam, wellington, managua, islamabad, muscat, oslo, abuja, niamey, victoria, rangoon, nouakchott, dakar, riyadh, belgrade.

This process is expected to take around 10 minutes.  During this time, the above servers will be completely offline.  Email and MySQL will continue to work during this downtime.

We sincerely apologize for the late notice on this maintenance!  Please contact Support with any questions you have!  We will update this post when the work is complete!

Update Sept 8th, 11:24pm PST: The maintenance has been completed and all affected servers are back online! Please contact Support if you have any issues that you feel may be related, or if you just have questions! -PatrickM

Previous updates available after the jump →


12:58 pm

‘Crate’ server down for RAID replacement (RESOLVED)

Posted (September 8th, 2011 at 12:58 pm PST) by danwy

The ‘crate’ server crashed due to a RAID card failure. We’re currently in the process of replacing the faulty hardware and should have the server back up within the next 15-30 minutes. We’ll update this status post as soon as we have any additional information.

Update Sept 8th, 1:25pm PST: We brought the server back up to investigate the possibility of it being a bad drive as opposed to the whole RAID card. Turns out it was the RAID card, so we’re rebooting it one more time now to replace it now. This should only take 15-20 minutes if everything goes well. We’ll update this post again then. -dan w

Update Sept 8th, 1:40pm PST: The server is back up and running now with the new RAID card. We tested several sites on the server across multiple Apache’s and everything seems to be fine on this server now. If you have any issues with a site hosted on this server, please contact support and they’ll be happy to troubleshoot for you. -dan w


10:47 am

Downtime on shared machine ” Dodgers ” (RESOLVED)

Posted (September 8th, 2011 at 10:47 am PST) by brians

The machine is offline at the moment. All connections ( http, ftp, sftp, ssh, etc ) will be unsuccessful until this issue has been resolved. Our admins are currently working towards getting the machine back online. We sincerely apologize for any inconvenience this has caused and appreciate your continued patience. Please check back for updates.

Update Sept 8th, 1:50pm PST: Our admin team has investigated this server and determined that there are no hardware issues with it. As the server has been stable since being rebooted we’re going to resolve this status post. We are, however, investigating other ways to improve stability on this server and will be implementing those shortly. If you have any further issues with sites on this server, please contact support for further troubleshooting. -dan w

Previous updates available after the jump →


10:08 am

Downtime on shared server “avarua” ( RESOLVED )

Posted (September 8th, 2011 at 10:08 am PST) by Justin@DH

We’re currently experiencing unscheduled downtime on avarua. Our techs are looking into the issue right now. We’ll post more updates when we have more information. Sorry about the inconvenience this is causing.

Update Sept 8th, 11:37am PST: All services have been restored. Long story short, a website on the machine was getting hammered by a msn bot. The user has been contacted regarding this matter. Sorry for any inconvenience this has caused. If you are still having problems, please submit a support request. – Brian S


 
© 1996-2011, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
10:03 am

Downtime on shared server “jagara” ( RESOLVED )

Posted (September 8th, 2011 at 10:03 am PST) by Justin@DH

We’re currently experiencing unscheduled downtime on jagara, and our techs are looking into it. We’ll post more updates when we have the specifics. Sorry about the inconvenience this is causing.

To see if you are on jagara (no other web servers, email or mysql servers are affected by this), check your FTP/Shell user’s machine. – Justin

Update Sept 8th, 11:02am PST: All services have been restored. Again, we apologize for any inconvenience that the downtime had caused. If you are still having issues, please submit a support request and we will gladly investigate your issue.


4:08 pm

Shared machine ” Manama ” is offline due to unscheduled filesystem check (RESOLVED)

Posted (September 6th, 2011 at 4:08 pm PST) by brians

Disk errors surfaced after the last reboot. A filesystem check is being performed at the moment. All services ( http, ftp, etc ) will remain offline until the filesystem check completes. We hope to have the machine back online shortly. We sincerely apologize for any inconvenience this has caused and appreciate your continued patience. Please check back for future updates.

Update Sept 6th 5:07pm PST: The server has started, and apaches are up and running. If you’re still seeing problems, please let us know ASAP. — Brian H


8:02 pm

VPS host homie-vserver102’s drive read only, needs a file system check. (RESOLVED)

Posted (September 5th, 2011 at 8:02 pm PST) by Brian@dreamhost

VPS host homie-vserver102 is currently having problems with one of its hard drives, and needs to run a check on it. That’s happening now, so we’ll keep this updated as the check progresses. Below is a list of affected VPS machines:

ps16218 ps16306 ps16359 ps16449 ps16783 ps16868 ps16893 ps17568 ps17652 ps18736 ps24610 ps35432 ps35589 ps36784 ps50395 ps54448 ps55561 ps16220 ps16315 ps16372 ps16515 ps16799 ps16872 ps16943 ps17576 ps17924 ps23969 ps26784 ps35506 ps35604 ps37078 ps51515 ps54577 ps56042 ps16223 ps16337 ps16407 ps16524 ps16829 ps16879 ps17531 ps17585 ps18279 ps24033 ps30609 ps35512 ps36160 ps38914 ps53549 ps54591 ps56981 ps16264 ps16342 ps16437 ps16553 ps16831 ps16890 ps17533 ps17588 ps18308 ps24560 ps33820 ps35520 ps39379 ps53728 ps55260

We apologize for the problems this causes and hope to have it taken care of soon!

Update Sept 6th: 3:25am: I am happy to report that the file system check has completed and the server is back up and running. This issue has now been resolved and your patience was greatly appreciated. If you have any unresolved issues, be sure to contact our tech support staff. – Chalise.

Previous updates available after the jump →


12:52 pm

Downtime on shared server ‘moran’ (RESOLVED)

Posted (September 5th, 2011 at 12:52 pm PST) by andrea

We’re currently experiencing unscheduled downtime on moran, and our techs are looking into it. So far it doesn’t seem hardware related. We’ll post more updates when we have the specifics. Sorry about the inconvenience this is causing.

To see you’re on moran (no other web servers, email or mysql servers are affected by this), check your FTP/Shell user’s machine. – Andrea

Update Sept 5th 3:52pm PST: We believe we found the reason the machine was instable earlier today and eliminated it. This issue is now resolved, thank you for your patience. If you continue experiencing problems with your websites, please contact support with the details, as it’s unrelated and we would be happy to help. – Andrea

Previous updates available after the jump →


4:39 am

Shared server ‘panamacity’ is down for raid card replacement (RESOLVED)

Posted (September 5th, 2011 at 4:39 am PST) by chalise

Users on the server ‘panamcity’ will be experiencing downtime with their server right now. Our admins report that the RAID card is causing problems and needs to be replaced. An admin is on the way to work on the machine; we will be updating this post as more information becomes available.

Update Sept 5th 6:45am PST: The panamacity webserver now has all of the problem disks replaced, a new raid card and new firmware on the raid card, which has proven in the past to be a more permanent fix for these issues with stability. The server is back up and we will be monitoring it. If you still see any issues after the reboots needed to get the firmware upgrade done, please contact support.


7:46 am

Shared server ‘wizz-ra’ is down for raid card replacement (RESOLVED)

Posted (September 4th, 2011 at 7:46 am PST) by Justin@DH

Users on the server tygra will be experiencing downtime with their server right now. Our admins report that the RAID card is causing problems and needs to be replaced. We’ve got an admin on the way to work on the machine and will be updating this post as more information becomes available.

Update Sep 4th 10:00am PST: The hardware issue with the raid card and a failed drive has been resolved, with both being replaced. wizz-ra is back up and running, if you are still seeing issues please contact support. – Justin


12:14 am

Downtime on homie-vserver102 due to unscheduled file-system check (RESOLVED)

Posted (September 3rd, 2011 at 12:14 am PST) by jj

The VPS host machine ‘homie-vserver1102′ has been rebooted into a read only file system and needs to have it’s file system checked. We’ve taken the machine off-line in order to ensure data integrity and will keep this post up to date as new information becomes available.

The following VPS services are currently being affected:

ps16218 ps16220 ps16223 ps16264 ps16306 ps16315 ps16337 ps16342 ps16359 ps16372 ps16407 ps16437 ps16449 ps16497 ps16515 ps16524 ps16553 ps16783 ps16799 ps16829 ps16831 ps16868 ps16872 ps16879 ps16890 ps16893 ps16943 ps17088 ps17531 ps17533 ps17568 ps17576 ps17585 ps17588 ps17652 ps17924 ps18279 ps18308 ps18736 ps23969 ps24033 ps24560 ps24610 ps26784 ps30609 ps33820 ps35506 ps35520 ps35561 ps35589 ps35604 ps36160 ps36784 ps37078 ps38914 ps39379 ps50395 ps51515 ps53549 ps53728 ps54448 ps54577 ps54589 ps54591 ps55260 ps55561 ps56042 ps56981

Update Sep 3rd 5:30am PST: The machine has finished the file system check and  is now back up and running. All VPS guests have been started up and this issue is now resolved. If you do have any problems with a VPS on the list please let support know. -JJ G Previous updates available after the jump →


3:51 pm

Shared server “mothman” being transferred to new hardware (RESOLVED)

Posted (September 2nd, 2011 at 3:51 pm PST) by Oscar

Due to a disk failure in the server’s raid array, and the continued failure during the rebuild process, the shared server mothman is being transferred over to new hardware. The old machine can not be brought up due to its degraded raid array and we are concerned that data may be lost if we rebuild it.

We’re currently restoring data to the new server from backups dated 2011-09-02. All mounted backups will have data from this date. During this process, you may update your files without risking them being overwritten by the data sync.

Please note, this post is related to data on the ‘mothman’ server only, not any other web server, mail server or database server. You can find your web server by checking on your ftp/shell user’s machine in the panel.

We will continue posting updates, on this page, throughout the process.

Update Sept 14th, 7:30pm PST:  The restore for mothman has now been completed! Horray! If you’re still missing data, please let our support staff know immediately. I’m leaving this post a sticky for the time being to make sure anyone possibly affected by this issue still sees it. –Brian H

Previous updates available after the jump →


3:47 pm

Scheduled Network Maintenace: Friday September 2, 2011 @ 10:00PM PST (RESOLVED)

Posted (September 2nd, 2011 at 3:47 pm PST) by brians

Our network engineers will be moving a distribution switch at our LAX datacenter. This will impact 3 racks of machines. They anticipate the process should be completed in 2 hours. Meaning it should be finished by Saturday September 3, 2011 @ 12:00AM PST.

Luckily there is a plan in place that should prevent any type of downtime. However, you just never know how these thing may go. Again, there are only 3 rows that will be affected if there is any type of downtime. I have posted the machines below.

If there is any downtime due to our scheduled maintenance, let me be the first to apologize. We will update the post once the maintenance has been completed. Please check back for updates.

chisinau homie-vserver172 homie-vserver174 asuncion homie-vserver173 peppy podgorica ulaanbaatar palikir mexicocity portlouis majuro valletta kualalumpur skopje homie-vserver220 bamako maputo rabat lilongwe melekeok dale siren babelfish slippy ecco starfox falco monaco andross dedede wario daisy nelly hippo hambone misha vamika jabberjaw flint spacey-vsql05 homie-vserver175 homie-vserver176 ljubljana yangon mogadishu pretoria apia kleeia kornephoros kochab homie-vserver226 homie-vserver225 homie-vserver224 kraz ksora kuma homie-vserver227 homie-vserver169 homie-vserver168 homie-vserver163 panamacity homie-vserver160 homie-vserver159 homie-vserver158 homie-vserver161 homie-vserver162 homie-vserver167 homie-vserver166 homie-vserver165 homie-vserver164 homie-vserver145 homie-vserver147 homie-vserver148 homie-vserver149 homie-vserver150 homie-vserver151 bucharest moscow kigali castries kingstown asuncion homie-vserver153 homie-vserver154 homie-vserver155 homie-vserver156 homie-vserver157 lisbon doha warsaw izar lima manila kathmandu abuja oslo belgrade riyadh dakar nouakchott rangoon victoria muscat managua wellington niamey windhoek amsterdam islamabad megalodon hammerhead nursehound mako homie-vsql44 homie-vsql43 knifetooth homie-vsql45 wobbegong swellshark nemo silvertip homie-vsql47 porbeagle chimaera homie-vsql46 wodnika stranger hauler sleeper treader aggregator miner shredder swashbuckler lax-tftp

Update Sep 3rd 12:00am PST: We did have some network disconnects during the window, however the work has been completed and the services listed above are all back and working as expected. If you do run into any problems with your sites on any of those machines please let support know. -JJ


9:37 am

Shared server ‘manila’ is down for hardware replacement (RESOLVED)

Posted (September 1st, 2011 at 9:37 am PST) by Justin@DH

Users on the server manila will be experiencing downtime with their server right now. Our admins report that there appears to be a memory issue, that is causing the server to reboot repeatedly. We have an admin on the case already, to replace the memory on the server. Should this not work, it will necessitate moving this server to new hardware. When we have more information we will update this post.

Update Sep 1st 1:30pm PST:The server is now back up and running. We tested an extensive sample of sites and everything appears to be back to normal. If you’re still having issues, please contact support with the details,as it’s most likely unrelated. We apologize again for the inconvenience this downtime has caused. – Andrea

Previous updates available after the jump →


 
© 1996-2011, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
5:45 am

Shared server ‘tygra’ is down for raid card replacement (RESOLVED)

Posted (September 1st, 2011 at 5:45 am PST) by jj

Users on the server tygra will be experiencing downtime with their server right now. Our admins report that the RAID card is causing problems and needs to be replaced. We’ve got an admin on the way to work on the machine and will be updating this post as more information becomes available.

Update Sep 1st 6:20am PST: Great news, the new card worked out just fine and the machine has been up and running without issue for the last 30 minutes. If you are having any problems with your sites running on ‘tygra’ please contact support. – JJ


10:38 am

MailMan discussion lists are broken ( RESOLVED )

Posted (August 31st, 2011 at 10:38 am PST) by brians

This only applies to lists hosted on a specific mailman instance. If the IP for your list sub-domain ( ex. lists.yourdomain.com ) resolves to ” 208.113.186.20 “, you are likely impacted. You can confirm whether you are affected by visiting the Domains::Manage section of the Panel and click the ” DNS ” link. Scroll down to the record for ” lists “.

More details will be provided once they are available. We hope to have this issue resolved shortly. Sorry for any inconvenience this has caused. Please check back for updates.

UPDATE August 31st 11:14AM PST:

The IP for the mailman instance was renumbered by mistake. The proper IP has been assigned to the mailman service. 66.33.216.202 is the correct IP that your list sub-domain need to resolve to. Unfortunately, this dns update needs to be manually performed. You can submit a support request and a member of our staff will gladly perform the dns update. Again, we sincerely apologize for any inconvenience this has caused and appreciated your continued patience. – Brian S


8:49 pm

Delays on all outgoing mail from web servers (RESOLVED)

Posted (August 30th, 2011 at 8:49 pm PST) by Jason C

Thanks to a pretty nasty looking attempted spam run across a number of our servers, customers may be seeing considerable delays with outgoing mail from their web scripts.

Not to worry though! Members of our Abuse and Admin teams are working together to stop the spammers in their tracks. Until they can, however, it might take a few hours for mail sent from your web server to reach its destination. We apologize for the inconvenience.

More updates will be coming just as soon as we have an idea of the scale of this attack.

Update Aug 31st 3:01am PST: Our admins report that mail is now doing better, while things may continue being slow for a few hours longer we are catching up on the backlog of messages and our admins will be keeping an eye on things throughout the day. – JJ

Update Aug 31st 9:01am PST: We’ve kept our eye on the outgoing mail relay server for several hours and it has been running smoothly. We’ve cleaned out all the mail initiated by the spam run and mail has been flowing smoothly since our last update. We’ve verified that all affected servers are running at their normal capacity and there have been no new support cases regarding this in over 6 hours, so we’re resolving this issue now. If you are still experiencing delayed mail, please contact support immediately and we’ll be happy to investigate it further for you. -dan w


7:20 pm

Shared MySQL server ‘daggernose’ downtime (RESOLVED)

Posted (August 28th, 2011 at 7:20 pm PST) by andrea

We’re currently running a file system integrity check on the shared mysql server ‘daggernose’. This should be a quick one, as there is not much data on the server. Sorry about the inconvenience this is causing, and we’ll update the post again soon. – Andrea

Update Aug 28th 7:33pm PST: Happy to report that the system check was successful and your MySQL server is up and running again. The initial higher load is due to some self-repairs that are going in the background, but it’s already coming down slow and steady. If you find any issues related to this, please contact support. – Andrea


12:19 pm

Shared server “ankara” down for file system check (RESOLVED)

Posted (August 27th, 2011 at 12:19 pm PST) by mir

The root file system on shared hosting server ankara is corrupt so we have had to take the server down for a file system check. We’ll provide more information as it becomes available. -John

Update Aug 28th 5:30pm PST: It’s been six hours now and the machine has remained steady. This issue is now resolved and if you happen to have any problems with your site on ‘ankara’ please let support know. — JJ G Previous updates available after the jump →


10:09 am

VPS host machine “homie-vserver76″ offline for network card replacement (RESOLVED)

Posted (August 27th, 2011 at 10:09 am PST) by mir

The network card on this host has gone bad and we are going to have to replace it. The estimated time to completion is 2 hours, here are the VPS guests on that server:

ps13053 ps13054 ps13099 ps13104 ps13110 ps13111 ps13112 ps13114 ps13115 ps13121 ps13122 ps13127 ps13130 ps13133 ps13209 ps13561 ps13601 ps13641 ps13650 ps13706 ps13707 ps13716 ps13717 ps13719 ps13753 ps13855 ps13888 ps13900 ps13912 ps13926 ps14003 ps14024 ps14084 ps14140 ps14160 ps14164 ps14177 ps14351 ps14379 ps15034 ps15168 ps26804 ps27807 ps27863 ps34611 ps34766 ps35240 ps35877 ps39612 ps39631 ps41337 ps41368 ps42410 ps44666 ps48393 ps50620 ps55618 ps55622 ps59585

We’ll keep this post updated and let you know as soon as we have been able to restore service. – John

Update Aug 27th 12:05pm PST: Thank you for your patience, the host is back up and guests are running so this issue is resolved. – John

Previous updates available after the jump →


2:49 am

Shared server “panthers” being transferred over to new hardware (RESOLVED)

Posted (August 27th, 2011 at 2:49 am PST) by jj

Due to a disk failure in the server’s raid array, and the continued failure during the rebuild process, the shared server panthers is being transferred over to new hardware. The old machine can not be brought up due to its degraded raid array and we are concerned that data may be lost if we rebuild it. The server is now running on new hardware and is mounting files from backup while we copy the data over locally. We will post updates, on this page, throughout the process.

Update Sept 5th 12:00pm PST: We’re happy to report that the restore is finally completed and this issue is resolved. We thank you for your patience! -John

Previous updates available after the jump →


11:34 am

Shared server ‘melekeok’ running from backups, being moved to new hardware, and file system check. (RESOLVED)

Posted (August 26th, 2011 at 11:34 am PST) by Brandon

The shared server ‘melekeok’ is currently restoring to new hardware using the last backups available.
We’ll keep this post up to date with current information as it becomes available.

Update Aug 30th 11:30pm PST: It looks like the restore has completed and all accounts have been moved over to the new hardware. If you have any further problems with your site on this server please let support know. – JJ

Update Aug 30th 4:20pm PST: We’re in the finish line; only a few accounts left to be restored to the new hardware. We’ll update this post again when we’re ready to switch back to local disks. – Andrea

Previous updates available after the jump →


8:03 pm

VPS sites down due to security update (RESOLVED)

Posted (August 25th, 2011 at 8:03 pm PST) by Oscar

If you are noticing that your VPS is up but your sites are currently down this is due to a security push-out done earlier today. We are aware of the issue and and our admin team is currently fixing it.  We are currently running an update to fix the issue with apache not being able to start. Once that update is done your sites will be back online.

Update Aug 26th, 3:08am PDT: The update was completed almost an hour ago. We were waiting to resolve this post to make sure no further issues came up. At this point, all downtime issues related to the apache update have been resolved. Should you have any further issues with your sites being down, please contact our support team and we will look into it for you. -Oscar

Previous updates available after the jump →


10:51 am

Shared server ‘panamacity’ offline due to RAM replacement. (RESOLVED)

Posted (August 25th, 2011 at 10:51 am PST) by Brandon

The shared server ‘panamacity’ has been taken offline while the RAM is replaced. Bad memory has been linked to recent instability on this server, and it should be more stable after this replacement. We’re currently expecting this to be completed within 30 minutes.

 

Update Aug 25th 11:30AM PST: The server is back online and running well. Should you be experiencing any issues, please contact our Technical Support staff!


 
© 1996-2011, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
1:10 am

Shared server ‘bujumbura’ offline for unscheduled file-system check (RESOLVED)

Posted (August 25th, 2011 at 1:10 am PST) by jj

The shared server ‘bujumbura’ is currently offline and requiring a file-system integrity check.  This process may take up to a couple hours to complete, but we will post regular updates here.

Update Aug 25th 8:15AM PST: All services are up and running so this matter is deemed to be resolved. Thank you for your patience! -John

Previous updates available after the jump →


6:37 pm

Firewall machines being upgraded tonight – Aug 8, 2011 (RESOLVED)

Posted (August 24th, 2011 at 6:37 pm PST) by Oscar

Tonight our admin team will be upgrading the core operating system on our internal firewall machines. We’ve been told that the method they’re using to do so should result in no downtime for our customers, but we still thought it was necessary to make you aware of the window in case something does happen to go awry.

We’ll be starting the upgrade tonight at 10:00pm PST and it should be completed by 12:00am PST.

Update Aug 25th 2:15AM PST: The firewall upgrade has gone through and was completed successfully around 10:30PM. This issue has now been resolved. -JJ


12:38 pm

Downtime on shared server ‘Blackhawks’ (RESOLVED)

Posted (August 24th, 2011 at 12:38 pm PST) by andrea

We’re currently experiencing some issues with some of the drives in your web server. The data volume appears to be intact, but we’re running an integrity check to make sure everything is up to par. We’ll post again very soon with an update to see if we could fix this up or if we need to migrate to new hardware. We’ll make sure that actual downtime is minimized as much as possible – Andrea

Update Aug 24th 1:47PM PST: The data check and reboot were successful, sites on ‘blackhawks’ are loading again fine. Sorry about the inconvenience this has caused!

Previous updates available after the jump →


10:17 pm

Shared servers suffering downtime from outgoing DDoS (RESOLVED)

Posted (August 23rd, 2011 at 10:17 pm PST) by patrickm

We are currently combating a rash of outgoing denial of service attacks coming from the following shared web hosting servers:

Abuja, Cebalrai, Cheleb, Diphda, Europa, Menkalinan, Mizar, Naos, Padres, Pagopago

A commonly exploited CMS script is being used to launch these attacks.  We are in the process of disabling these scripts to halt the attacks and return the servers to normal operation.  If you receive a security notification from us, here’s some fresh new info on how to respond: http://blog.dreamhost.com/2011/08/23/about-dreamhost-security-notifications/

We will update this post with more details!

Update: This issue was resolved shortly after we posted about this downtime! Our apologies for not updating this earlier this morning! If you have any questions about this downtime, or security on your account, please contact Support! Thanks!


3:10 pm

Downtime on shared web server Panamacity (RESOLVED)

Posted (August 22nd, 2011 at 3:10 pm PST) by andrea

Sorry about the current issues you’re experiencing with your sites on Panamacity. Our technicians are investigating the cause of the instability. We’ll update this post as soon as we have news on its status. – Andrea

LAST UPDATE 2011-08-22 4:56PM PST: We have it confirmed that the machine is now back up and serving sites. We didn’t have to migrate to new hardware. Sorry again about the inconvenience this has caused. If you’re still having issues, please contact support with the details. – Andrea

UPDATE 2011-08-22 4:02PM PST: We were unable to fix the machine remotely, and someone is now on his way to the datacenter to take care of it manually. The data may need to be migrated to new hardware, but we’ll let you know if and when that happens. We appreciate your continued patience while we try to fix this server. – Andrea


1:43 pm

VPS host machine “homie-vserver223″ offline for unscheduled file system check ( RESOLVED )

Posted (August 22nd, 2011 at 1:43 pm PST) by brians

A couple of the drives in the RAID array went into read-only mode. A filesystem check on the drives is being performed at the moment. As a result of the FSCHK, all VPS guests on this host will remain offline until the filesystem check completes. The affected guests are listed below. Unfortunately we do not have an ETA at this time. We sincerely apologize for any inconvenience this has caused and appreciate your continued patience. Please check back for future updates.

ps46972 ps46990 ps53517 ps56727 ps56838 ps56841 ps56873 ps56979 ps57011 ps57126 ps57129 ps57194 ps57217 ps57218 ps57281 ps57307 ps57317 ps57319 ps57365 ps57373 ps57399 ps57415 ps57417 ps57418 ps57421 ps57422 ps57428 ps57438 ps57469 ps57498 ps57517 ps57596 ps57716 ps57749 ps57902 ps57959 ps58248 ps58346 ps58497 ps58506 ps58514 ps58899 ps58913 ps58969 ps59330 ps59401 ps59426 ps59513 ps59604

UPDATE 2011-08-22 4:25PM PST: All phases of the filesystem check have completed. The VPS guests are starting as I update this post. Again, we sincerely apologize any inconvenience this has caused and appreciate your continued patience. If you are still having problems, please submit a support request. – Brian S

Previous updates available after the jump →


1:07 pm

Downtime on homie-vserver225 – RESOLVED

Posted (August 21st, 2011 at 1:07 pm PST) by andrea

LATEST UPDATE 2011-08-21 8:11PM PST: The data integrity check is now finished on your server and your sites should be loading again. We apologize for the lengthy downtime, and please let support know if something is still wrong with your VPS or individual sites, as it’s most likely unrelated. – Andrea

Unfortunately, after rebooting homie-vserver225, it didn’t come back as expected. We’re running a data integrity check right now, and will reboot again as soon as it’s done. We’ll update this post with any new details we learn from the case. If you’re on the following VPS, this is why you may be affected right now:

ps25286 ps37472 ps47015 ps47018 ps47024 ps47025 ps47026 ps47035 ps47037 ps47038 ps47039 ps47055 ps47068 ps47069 ps47071 ps47077 ps47095 ps47128 ps47143 ps47150 ps47171 ps47237 ps47252 ps47272 ps47365 ps47384 ps47393 ps47395 ps47430 ps47464 ps47500 ps47591 ps47669 ps47747 ps47752 ps47755 ps47791 ps47865 ps47869 ps47875 ps47896 ps48139 ps48192 ps48202 ps48203 ps48241 ps48288 ps48294 ps48316 ps48325 ps48470 ps48478 ps48732 ps48841 ps48847 ps49156 ps49337 ps49503 ps49643 ps49827 ps50119 ps50290 ps50622 ps50623 ps52252 ps52458 ps54138

Previous updates available after the jump →


12:19 am

VPS host machine “homie-vserver166″ offline for unscheduled file system check (RESOLVED)

Posted (August 20th, 2011 at 12:19 am PST) by jj

The VPS host machine “homie-vserver166″ is currently executing an unscheduled file system check in order to ensure data integrity. We will keep this post up to date as new information becomes available.

List of affected VPS guests:
ps23438 ps23468 ps23561 ps23597 ps23675 ps23713 ps23834 ps24127 ps24259 ps24337 ps24380 ps24557 ps24821 ps24863 ps24902 ps24960 ps25031 ps25103 ps25247 ps25373 ps25685 ps26641 ps26645 ps26647 ps27966 ps28153 ps28787 ps28835 ps29210 ps29220 ps29246 ps29561 ps30192 ps30230 ps30232 ps30237 ps30691 ps30726 ps30907 ps33516 ps33550 ps34083 ps34098 ps34447 ps35551 ps37059 ps41610 ps47203 ps47571 ps47712 ps53324 ps53362 ps53476 ps54381 ps54427 ps54450

Update Aug 20th, 5:08am PDT: The file system check as completed and the host machine, as well as the VPS services, are back up and running correctly. If you still have any problems please let support know. -JJ


7:11 am

mod_xsendfile disabled temporarily to preserve Apache function (RESOLVED)

Posted (August 19th, 2011 at 7:11 am PST) by mir

We’re seeing apache errors come up due to a change to mod_xsendfile which appears to have been triggered by a Development Team update to that module. As a result we are not able to complete apache configurations on some machines. We’re in the process of rolling back the change and fixing affected apache instances. We’ll update this post as we have more information.

UPDATE 2011-08-19 04:05PM PST: We resolved this at around 3:00PM today. The changes we rolled back are back in place on the machines they should be on. We are in the clear. -Ralph

Previous updates available after the jump →


6:19 am

Homie-vserver175 going offline for unscheduled file-system integrity check ( RESOLVED )

Posted (August 19th, 2011 at 6:19 am PST) by patrickm

We are taking the Vserver host ‘homie-vserver175′ offline temporarily for an unscheduled file-system integrity check.  We expect this process to take a couple hours to complete. This means the following VPS guests are offline during this check:

ps23515 ps26680 ps26786 ps29036 ps33700 ps33725 ps33757 ps33779 ps34049 ps34215 ps34403 ps34417 ps34438 ps34449 ps34450 ps34457 ps34481 ps34482 ps34494 ps34504 ps34541 ps34644 ps34712 ps34714 ps34732 ps34954 ps35115 ps35133 ps35160 ps35313 ps35500 ps36141 ps36213 ps36313 ps36738 ps37748 ps37755 ps39117 ps39562 ps39772 ps42478 ps43903 ps44577 ps45233 ps46387 ps46597 ps46647 ps46793 ps46961 ps46985 ps47375 ps48810 ps48990 ps48992 ps49128 ps49343 ps50681 ps52449 ps56463 ps56777 ps57623 ps57642 ps58474

Please watch this space for updates as the process moves forward! Feel free to contact Support with any questions you may have, as well!

UPDATE 2011-08-19 3:45PM PST: The filesystem check has completed. The VPS guests are starting as I type this status update. Sorry again for the inconvenience this has caused. If you continued to notice problems, please feel free to submit a support request. – Brian S

Previous updates available after the jump →


 
© 1996-2011, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
2:18 pm

Platy MySQL Server Downtime (RESOLVED)

Posted (August 16th, 2011 at 2:18 pm PST) by Justin@DH

The mysql server platy stopped responding, and our datacenter team are on the job to see why it is not responding to remote powercycle. It should be a quick fix to get it rebooted and running again, but should it be a less easy job, we will update this status post as such. Our apologies for the downtime related to this issue.

Update 2001-08-16 14:39 PDT – Platy has been rebooted, and the cause found for its not being remote powercycle-able. Our admins will be working on getting that fixed up. We are watching the platy server to make sure it keeps running normally.


2:23 pm

VPS upgrades to Debian OS Squeeze – limited release (RESOLVED)

Posted (August 15th, 2011 at 2:23 pm PST) by andrea

To follow up on the recent vserver host upgrades to the latest Debian OS ‘Squeeze’, we’ll start upgrading the guest servers on those hosts, one by one. We’ll commence the process later today.
At this time, the upgrade will only occur on VPS guests that do not have admin users on them. This is to preserve the integrity of any custom installed programs. If you have an admin user and would like to be upgraded right now, please contact support directly through the panel.

The upgrade process itself only takes 5-20 minutes to do per guest, depending on the size of their home directories. During this time, you may experience some downtime while the files are upgraded and services are reconfigured. Unfortunately, we’re unable to tell exactly how long each upgrade will take and as such, we’re unable to give a valid estimate on when the whole process will finish. We appreciate your understanding and stand by if you have any questions.

To check if you’re upgraded yet, connect to your shell and run the following command:

cat /etc/debian_version

If you see 6.0.2, you’re upgraded!  If not, sit tight, and it will be all over and better soon. We appreciate your understanding and patience during this process.

Update 2011-11-28 10:30am: We will not be performing any more upgrade during the holidays but will start back up with these for next year. We will resolve this status post for now. – Justin


11:50 am

Shared server “earthquakes” online and restoring to replacement hardware (RESOLVED)

Posted (August 15th, 2011 at 11:50 am PST) by danwy

The shared web server ‘earthquakes’ was brought back online following a successful file-system check.  Due to recent issues with the RAID controller in the current server, we are restoring the data for this server on to completely new hardware.  When that restore is complete, we will sync the latest data from the current server over to the new one and bring it online as the live server.

During this process, we will update on the progress of the restoration to the new hardware and the final sync process. Check back on this post for the latest ETA on the final switch-over to avoid unnecessary data loss during the transition.

Update Aug 16th, 2:00am PDT: The restoration process on the new hardware for the server is running smoothly. Since we are not restoring to the live server, the process should be quicker and will not cause any downtime for the live server. Due to this, our updates on the process will be less frequent as the new hardware is readied. Please contact Support with any questions you have about this! -PatrickM

Previous updates available after the jump →


11:46 am

Maintenance on dreamhoststatus.com tomorrow (Resolved)

Posted (August 15th, 2011 at 11:46 am PST) by Terri

We will be switching dreamhoststatus.com tomorrow to another server, starting at around 11 a.m. PDT. Please pardon our dust as we make the switchover to make sure our status site is the best it can be!


8:52 pm

Scheduled maintenance for homie-vsql89 on Sunday, August 21st at 9pm PDT (RESOLVED)

Posted (August 14th, 2011 at 8:52 pm PST) by Brian@dreamhost

On Sunday, August 21st we’ll be taking homie-vsql89 down for approximately 20 minutes to maintain the server’s backplane connections. The following servers will be affected:

psmysql58358 psmysql58375 psmysql58382 psmysql58422 psmysql58437 psmysql58438 psmysql58447 psmysql58448 psmysql58455 psmysql58456 psmysql58473 psmysql58481 psmysql58523 psmysql58535 psmysql58552 psmysql58553 psmysql58850 psmysql58876 psmysql58887 psmysql58918 psmysql58940 psmysql58959 psmysql58974 psmysql59083 psmysql59089

We apologize for any inconvenience this may cause!

Update Aug 21st, 9:28pm PDT: Our admins have let us know that they have finished the scheduled maintenance. If anyone notices issues with their MySQL VPS (listed above) please contact our support team ASAP.


10:33 am

VPS host machine “homie-vserver196″ offline for File System Check (RESOLVED)

Posted (August 13th, 2011 at 10:33 am PST) by Oscar

The VPS host machine “homie-vserver196″ rebooted into a read only file system and needs to have it’s file system checked. We’ve taken the machine off-line in order to ensure data integrity and will keep this post up to date as new information becomes available.

List of affected VPS guests:

ps33593 ps33904 ps34490 ps34610 ps34657 ps34772 ps34792 ps34806 ps34854 ps35101 ps35250 ps35333 ps35694 ps35797 ps36205 ps36290 ps36358 ps36449 ps36534 ps36721 ps36727 ps36786 ps36997 ps37034 ps37225 ps37323 ps37334 ps37695 ps37709 ps37734 ps37789 ps38309 ps58379 ps58609 ps58826 ps58894

UPDATE Aug 14th, 3:20AM PST: The file system check has finished and we’ve been watching the machine for the last hour and a half, everything seems to be running properly now. If you still have any problems with your VPS running on homie-vserver196 please contact support. –JJ Previous updates available after the jump →


11:40 am

Outgoing DDOS Causing Instablility ( RESOLVED )

Posted (August 12th, 2011 at 11:40 am PST) by brians

This is only impacting a specific row of machines. Our admins are diligently working towards resolving this matter. We are only noticing issues on the following machines at this moment:


dubhe diphda haldus jaga edasich, europa hassaleh eltanin markab cheleb cebalra mizar eltanin corcaroli

We believe that we have things under control. However, you may notice periods of connection loss / failure to your website. We will update the post once the outage has been fully resolved. We sincerely apologize for any inconvenience this has caused and appreciate your continued patience. Please check back for future updates.

UPDATE 2011-08-12 3:03PM PST: The DDOS is no longer occurring. The DDOS was caused by users running outdated / exploitable software. The problematic users have been contacted regarding this mater. If you are still having problems, please contact tech support. – Brian S


10:52 am

Scheduled Maintenace on Various Domain TLDs

Posted (August 12th, 2011 at 10:52 am PST) by brians

Our affiliate will be performing maintenance from Saturday August 13, 2011 4:00PM PST thru Sunday Aug 14, 2011 2:00AM PST. You can learn more here:

http://forums.myorderbox.com/index.php?/topic/1111-scheduled-maintenance-on-orderbox

Due to the maintenance, you will not be able to update, register, renew the domain registrations for TLDs pasted below:

.co.uk, .ca, .us, .biz, .es, .me, .in, .tv, .eu, .co.nz, .de, .org.uk, .co, .ws, .mobi, .name, .cc, .asia, .me.uk, .com.co, .org.nz, .net.nz, .bz, .mn, .net.com, .nom.co

This only applies to domain registrations. Your existing domains will still function. Meaning, all services ( web, mail, etc ) will continue to work.

So if you need to make any modifications to your domain registration, please make them before August 13, 2011 4:00PM PST. You’ve been warned :)

Update: This maintenance is now complete.  Please report any issues you see to Support!


4:41 am

Shared server “lansky” online while restoring to new hardware (RESOLVED)

Posted (August 11th, 2011 at 4:41 am PST) by patrickm

The shared server ‘lansky’ did not pass a file-system integrity check, causing our Admin team to replace the hardware for this server. We have mounted the backup server such that all sites hosted on this server are online while we restore the backup server’s data on to the new hardware.

During this process we are constantly monitoring the server for high load due to the data transfer and will work to keep that under control to provide the most stability possible until this is complete.

We will update this post regularly with updates on the progress of the restoration. Please check back here for that and contact Support with any questions you have!

Update Aug 18th, 5:20pm PDT: All restores are now complete. Please check your sites and if you notice any problems or any missing files please contact our support staff. We are keeping the old machine up for a while in case you notice any files that may not have copied over. We are confident that we got everything off of the old machines but we are more than happy to assist you further. Our sincere apologies as a company and my sincere apologies as the VP of Customer Experience. I understand that this issue was a nightmare for many of you and we have started to put together some steps that will help us deal with situations like this better in the future. Number one being communication to not only you our loyal customers but within the company to truly spread the severity of the issue throughout the company. Getting our bright minds together to come up with creative solutions has been our companies identity and I apologize for the failure on our part with this particular incident. Thank you all for the patience you have been able to give us during this downtime issue. We will do our best to make sure that we keep you informed in a timely manner as well as come up with creative solutions to difficult problems.

Previous updates available after the jump →


2:48 pm

VPS host machine “homie-vserver148″ offline for File System Check (RESOLVED)

Posted (August 10th, 2011 at 2:48 pm PST) by Brandon

The VPS host machine “homie-vserver148″ rebooted into a read only file system and needs to have it’s file system checked. We’ve taken the machine off-line in order to ensure data integrity and will keep this post up to date as new information becomes available.

List of affected VPS guests:

ps23767 ps24054 ps24322 ps24555 ps24790 ps24879 ps24983 ps25006 ps25306 ps25634 ps25984 ps25995 ps26000 ps26087 ps26418 ps26556 ps26578 ps26704 ps28734 ps28772 ps28847 ps28903 ps28952 ps28973 ps28999 ps29047 ps29107 ps29718 ps30024 ps30079 ps30470 ps34080 ps35093 ps35888 ps36003 ps36051 ps37230 ps37353 ps37370 ps37383 ps37418 ps38721 ps38725 ps38852 ps38877 ps41102 ps42055 ps42056 ps48538 ps49132 ps52528 ps54100 ps54625 ps55489

Update Aug 11th, 10:15am PDT: The file system check has completed, and VPS are running again. If you still see issues, please contact support. – Justin

Previous updates available after the jump →


 
© 1996-2011, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
2:44 pm

Stability issues on shared webserver “melekeok” (RESOLVED)

Posted (August 10th, 2011 at 2:44 pm PST) by Brandon

We’re currently investigating the cause of stability problems on the shared web server “melekeok”. We’ll update this post when more information becomes available.

Update Aug 10th, 5:20pm PDT: Our admin team has been looking into this and it would appear that some of the drives in the machine went bad. We’re working on replacing them now. – Jason

Update Aug 11th, 7:00am PDT: The melekeok server has a 17 hour uptime now and a normal load, we believe the issues with this server are resolved. If you have further problems please contact support. – Justin


7:31 am

Recent AutoPay Payments Delayed (RESOLVED)

Posted (August 10th, 2011 at 7:31 am PST) by Justin@DH

We are seeing customers reporting that autopay has not been automatically charging them if their appropriate payment date was in the last day. We are having our development team look into this issue now. When we have further info we will update this post.

Update: According to our developers, it seems that our automatic billing process did not properly run one day earlier this month. The process is running now to take care of the users who were not charged. If you still notice any issues with autopay or have any questions or concerns, please contact support.


7:36 am

Shared server “turmagar” suffering from RAID card issues ( RESOLVED )

Posted (August 9th, 2011 at 7:36 am PST) by Justin@DH

Users on the shared server “turmagar” may be noticing some issues getting thru to their server right now. According to our admin team the RAID card appears to be acting up and we are working on getting it replaced. When we have further information or an ETA we will update this post. Our apologies for the inconvenience caused by this issue.

UPDATE 2011-08-09 12:05PM PST:

The RAID card has been replaced. All services on the machine are fully functional. We sincerely apologize for any inconvenience this has caused and appreciate your continued patience. If you are still noticing problems, please contact support. It’s possible that the downtime that you experiencing is not related to this status post. -Brian S.


6:50 am

Shared server “earthquakes” offline for file system check ( RESOLVED ).

Posted (August 9th, 2011 at 6:50 am PST) by Justin@DH

The shared web server “earthquakes” is currently off-line running a file system check. We will keep this post up-to-date with it’s progress as further information becomes available.

UPDATE 2011-08-09 12:11PM PST: The filesysten check has completed and the machine is back online. Sorry for the inconvenience the downtime had caused. We really appreciate your patience. If you are still having problems, please submit a support request. More than likely the problems that you’re encountering aren’t related to this status post. -Brian S.


6:49 am

Shared server “nayda” suffering from RAID card issues ( RESOLVED ).

Posted (August 9th, 2011 at 6:49 am PST) by Justin@DH

Users on the shared server “nayda” may be noticing some issues getting thru to their server right now. According to our admin team the RAID card appears to be acting up and we are working on getting it replaced. When we have further information or an ETA we will update this post. Our apologies for the inconvenience caused by this issue.

UPDATE 2011-08-09 12:10PM PST:

The RAID card has been replaced. All services on the machine are fully functional. We sincerely apologize for any inconvenience this has caused and appreciate your continued patience. If you are still noticing problems, please contact support. It’s possible that the downtime that you experiencing is not related to this status post. -Brian S.


3:03 pm

Shared server “lacerta” suffering from RAID card issues. (RESOLVED)

Posted (August 8th, 2011 at 3:03 pm PST) by Jason C

Users on the shared server “lacerta” may be noticing some issues getting thru to their server right now. According to our admin team the RAID card appears to be acting up.

Our team is working on fixing things up for users right now and hopefully we’ll have some good news shortly. Keep an eye on this post for updates!

Update Aug 8th, 5:10pm PDT: Our team has now replaced the RAID card in the server and the machine is back up and chugging along. We’ll continue to keep an eye on things to make sure nothing goes crazy, but we’re considering this one resolved. – Jason

Update Aug 8th, 3:35pm PDT: The RAID card has been verified as being bad, and our admin team is currently en route to the data center in order to replace the faulty hardware. We’ll update this post with further information as it becomes available. -Brandon


2:44 pm

Shared server “angel” offline for file system check. (RESOLVED)

Posted (August 8th, 2011 at 2:44 pm PST) by Brandon

The shared web server “angel” is currently off-line running a file system check. We will keep this post up-to-date with it’s progress as further information becomes available.

Update Aug 8th, 9:00pm PDT: The file system check has been competed and our admin team has brought up the Apache services. We’ll continue to monitor the server, but customers on angel should now see their sites without any issues. If you do notice something out of the ordinary, however, please let our support team know! – Jason

Previous updates available after the jump →


1:48 pm

Problems with the Dreamhost Web Panel – RESOLVED

Posted (August 8th, 2011 at 1:48 pm PST) by andrea

We’re currently experiencing some issues with the DreamHost Web Panel which causes slowness or inaccessibility. We’re working on resolving this. Sorry about the inconvenience this is causing. We’ll update this post as soon as we have more information.

Update Aug 8th, 3:46pm PDT: We’re still working on stabilizing the panel. We appreciate your patience! Will post again when we have more details on the progress.

Update Aug 8th, 3:25pm PDT: We made some configuration changes that resolved the earlier database issues that caused the slowness and downtime. We already noticed significant improvement in speed and stability, but will keep an eye on it for a while. Thanks again for your understanding while we worked on it!


1:27 pm

VPS host homie-vserver160 downtime for restore from backups (RESOLVED)

Posted (August 8th, 2011 at 1:27 pm PST) by Justin@DH

The VPS host server homie-vserver160, which previously had a file system check, appears to have worse issues after the file system check that is requiring us to restore from backups to fix VPS on it that are not properly starting. The admin team is working on this right now.

The following VPS machines are affected:

ps23389 ps23462 ps23778 ps24415 ps24530 ps24914 ps24976 ps25300 ps25380 ps25466 ps26543 ps26795 ps27767 ps28063 ps29181 ps35894 ps37992 ps23423 ps23635 ps24163 ps24490 ps24558 ps24921 ps25021 ps25302 ps25401 ps25879 ps26659 ps27665 ps27821 ps28886 ps29267 ps36271 ps39870 ps23432 ps23669 ps24210 ps24509 ps24794 ps24927 ps25028 ps25357 ps25431 ps26114 ps26774 ps27670 ps27982 ps29073 ps35496 ps36645 ps41116

Update Aug 8th, 9:20pm PDT: We’re back! All customers on this VPS host server should be back up and running now. If you happen to notice anything out of the ordinary with your site, please don’t hesitate to contact our support team immediately. – Jason


12:36 am

Shared server ‘perseus’ offline for unscheduled hardware replacement (RESOLVED)

Posted (August 8th, 2011 at 12:36 am PST) by patrickm

We regret to report that the shared server ‘perseus’ had to be taken offline to perform a file-system integrity check, as it came back from a reboot with a read-only file-system too many times.  Unfortunately, this process did not work and we are now replacing the hardware for this server and once that is complete, we will be restoring the server from backups.  Once this begins, the server will remain online during the restore.  We will update this post with details!

Update Aug 8th, 8:01pm PDT: All of the Apache services are back up and the server has been working without issues for the past hour now. If you happen to notice any further issues, please contact our support team. – Jason

Previous updates available after the jump →


 
© 1996-2011, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
4:11 pm

Shared server “turmagar” down with RAID card issues. (RESOLVED)

Posted (August 7th, 2011 at 4:11 pm PST) by Jason C

Customers on the server “turmagar” may notice an issue with their sites. It would appear that the RAID card in the server has up and failed and one of our admin team members is on their way to the datacenter now. We will do everything we can to keep this post updated with details as the matter progresses.

Update August 7th, 6:04pm PDT: Our admin team has let us know that the server is back up and has been holding steady for the past 15 to 20 minutes. We’ll be keeping an eye on it to make sure it stays that way, but if you happen to see any issues, please don’t hesitate to contact support. – Jason


6:24 am

VPS host homie-vserver160 downtime for file system check (RESOLVED)

Posted (August 7th, 2011 at 6:24 am PST) by Justin@DH

The VPS host server homie-vserver160 is currently offline for file system check. We will update you through this post as the file system check runs.

The following VPS machines are affected:

ps23389 ps23462 ps23778 ps24415 ps24530 ps24914 ps24976 ps25300 ps25380 ps25466 ps26543 ps26795 ps27767 ps28063 ps29181 ps35894 ps37992 ps23423 ps23635 ps24163 ps24490 ps24558 ps24921 ps25021 ps25302 ps25401 ps25879 ps26659 ps27665 ps27821 ps28886 ps29267 ps36271 ps39870 ps23432 ps23669 ps24210 ps24509 ps24794 ps24927 ps25028 ps25357 ps25431 ps26114 ps26774 ps27670 ps27982 ps29073 ps35496 ps36645 ps41116

Update August 7th, 2:00pm PDT: The file system check has successfully completed, and all VPS guests are back online! Should you still be experiencing any issues with your VPS guests or domains, please submit a ticket to our helpful Technical Support team! -Brandon

Previous updates available after the jump →


8:11 pm

Shared server “cebalrai” is currently down due to RAID card issues (RESOLVED)

Posted (August 6th, 2011 at 8:11 pm PST) by Brian@dreamhost

Unfortunately, the server “cebalrai” has crashed due to a bad RAID hardware controller. Our datacenter techs are scrambling to get this sorted out, but for the time being, the machine is down.

We’ll update this post as things progress, and apologize for the inconvenience.

UPDATE: 2011-08-07 6:30AM PST: No more problems seem to be coming up here, if you do still have any issues with sites on this machine please let support know. –JJ

UPDATE: 2011-08-07 05:25AM PST: The machine has been up for almost two hours at this point and everything seems to be running smoothly. We’ll keep an eye on the machine and close up this post after we’re sure it is in the clear. –JJ

UPDATE: 2011-08-07 02:45AM PST: Our admins are still on this issue, the server problems continue and we will post updates when more information becomes available. –JJ

UPDATE: 2011-08-06 10:08PM PST: Spoke a little too soon, and it seems the server is still having problems. We’re still investigating, and will report when we have more info –Brian H

UPDATE: 2011-08-06 09:50PM PST: Ok, the server should be up now, and it starting its services. Domains should be backup within the next few minutes. If you don’t see your domain start up, please let support know ASAP –Brian H

UPDATE: 2011-08-06 09:25PM PST: The RAID card is now being replaced. We should have the server back up within the next 30 minutes or so. –Brian H


1:46 pm

Changes made through Web Panel delayed (RESOLVED)

Posted (August 5th, 2011 at 1:46 pm PST) by patrickm

We are currently investigating an issue which is causing any changes to your services made within the Web Panel to be delayed.  Most notably this is causing delays with the creation of MySQL Databases, adding and removing User accounts and installing OneClick Installs.  However, this is not limited to those tasks, as most changes are delayed at this time.

We will update this post with more details on this issue shortly!

UPDATE: 2011-08-06 10:30PM PST It looks like these delays are well taken care of at this point, so this issue seems to be resolved. If you have issues with long delays with making changes in the control panel, please let support know. -JJ Previous updates available after the jump →


5:11 pm

Emergency datacenter maintenance tonight – Aug 4th, 11pm (PST) (RESOLVED)

Posted (August 4th, 2011 at 5:11 pm PST) by danwy

We just received notice from one of our datacenters that they will be performing emergency maintenance tonight, Aug 4th at 11pm (PST). They expect everything to to function normally during the maintenance window, but we wanted our customers to be informed of the work that was being done. If any issues arise as a result of the work we will update this post. Here is the notice that was sent to us:

An emergency maintenance on Alchemy’s downtown switching and routing equipment will be performed during a scheduled maintenance window on Thursday, August 4th, 2011. This window will start Thursday August 4th at 2300 and end Friday August 5th at 0500 (time local to the device). During this maintenance, all traffic will be automatically rerouted through a redundant link.

The maintenance should be transparent to you and no interruptions of service are anticipated; however as with any maintenance, there is a small possibility of service interruptions as configuration changes are made. During the maintenance window we expect the duration of the work to take approximately two (2) hours.


12:07 pm

VPS host homie-vserver150 offline (RESOLVED)

Posted (August 4th, 2011 at 12:07 pm PST) by Brandon

The VPS host machine “homie-vserver150″ is currently offline. Our admin team is troubleshooting several variables on the machine that came our way with the recent Debian Squeeze upgrade. We do not currently have an estimated time until completion for this, but will update this status post as soon as we have further information.

The following VPS guests are affected:

 

ps23581 ps23615 ps23992 ps24245 ps24324 ps24547 ps24753 ps24853 ps24866 ps24924 ps24971 ps24987 ps25148 ps25272 ps25387 ps25414 ps25464 ps25567 ps25752 ps25782 ps26504 ps28749 ps28885 ps29179 ps29302 ps29347 ps29451 ps29687 ps29689 ps29702 ps29770 ps29932 ps29960 ps30418 ps30465 ps30780 ps30787 ps30800 ps30847 ps30851 ps31117 ps34996 ps35471 ps36428 ps36923 ps42228 ps44130 ps44592 ps45254 ps46631 ps47027 ps47672 ps48039 ps48051 ps48053 ps58256

 

UPDATE Aug 4th 12:26PM PST: The guests on this machine are now back online, and the Squeeze kinks ironed out. If you’re still seeing any issues, please contact our Technical Support team! -Brandon


3:34 am

VPS host homie-vserver165 downtime for file system check ( RESOLVED )

Posted (August 4th, 2011 at 3:34 am PST) by jj

The VPS host server homie-vserver165 is currently offline for file system check. We will update you through this post as the file system check runs.

The following VPS machines are affected:

ps23353 ps23467 ps23674 ps23712 ps23744 ps23783 ps23833 ps23957 ps24126 ps24168 ps24842 ps24858 ps24929 ps25011 ps25355 ps25439 ps25528 ps25541 ps25719 ps25798 ps25816 ps25980 ps26298 ps27366 ps27745 ps28430 ps28453 ps28454 ps28538 ps28678 ps29032 ps29224 ps29237 ps29634 ps29798 ps29801 ps29904 ps30034 ps30102 ps30555 ps30649 ps31151 ps31204 ps33408 ps33646 ps35337 ps35557 ps35941 ps38178 ps39342 ps42899 ps44446 ps44802 ps44847 ps45047 ps45062 ps45100 ps45836 ps46182 ps46255 ps46644 ps46947 ps47470 ps47521 ps47853 ps52325 ps52513 ps52671 ps53176 ps53550 ps54324 ps55467 ps55480 ps55796

UPDATE Aug 4 2011 12:48PM PST: The filesystem check has finished. Again, we sincerely apologize for any inconvenience this has caused and really appreciate your patience. If you are still experiencing problems, please contact tech support. We will gladly take a closer look at your specific server.

Previous updates available after the jump →


10:44 am

Shared server achird down due to RAID hardware issues (RESOLVED)

Posted (August 3rd, 2011 at 10:44 am PST) by Brandon

The shared webserver “achird” is offline due to a RAID card issue. Our admin team is currently en route to the datacenter to replace the faulty hardware. The current estimated time until completion is one hour.

 

Update August 3rd, 11:45am PDT: We’ve now replaced the faulty RAID card in this server and it is back online. Should you still be experiencing issues with your sites, please contact our Technical Support staff. -Brandon


9:57 am

Shared server ljubljana down due to RAID hardware issues (RESOLVED)

Posted (August 3rd, 2011 at 9:57 am PST) by mir

The server ljubljana is currently down due to issues with its RAID card. The admin team is investigating and we’ll provide more information here as it becomes available. -John

Update August 3rd, 11:23am PDT: We’ve now replaced the faulty RAID card in this server and it is back online. Should you still be experiencing issues with your sites, please contact our Technical Support staff. -Brandon Previous updates available after the jump →


2:42 pm

VPS host homie-vserver76 offline due to stability issues (RESOLVED)

Posted (August 2nd, 2011 at 2:42 pm PST) by Brandon

Our admin team is investigating the cause of today’s stability issues on the machine “homie-vserver76″. This machine is currently offline. We will update this post with more information as it becomes available.

Here is a list of affected VPS guests:

 

ps13053 ps13054 ps13057 ps13099 ps13104 ps13110 ps13111 ps13112 ps13114 ps13115 ps13119 ps13121 ps13122 ps13127 ps13130 ps13133 ps13209 ps13561 ps13601 ps13641 ps13650 ps13706 ps13707 ps13716 ps13717 ps13719 ps13739 ps13753 ps13855 ps13888 ps13900 ps13912 ps13926 ps14003 ps14024 ps14084 ps14140 ps14160 ps14164 ps14177 ps14347 ps14351 ps14379 ps15034 ps15168 ps26804 ps27807 ps27863 ps34611 ps34766 ps35240 ps35877 ps39612 ps39631 ps41337 ps41368 ps42410 ps44666 ps48393 ps50620 ps55068 ps55618 ps55622 ps55625

 

UPDATE 2011-08-02 3:45pm PST: We’ve isolated and resolved the issue on this machine. It seems that a recent bug in our Squeeze implementation forced this machine to boot into the wrong runlevel. This bug has been fixed, but the fix had not yet been applied to this machine. It’s now been applied, and the machine is back online with all VPS guests starting now.


 
© 1996-2011, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
2:26 pm

Shared server ‘bulls’ offline for unscheduled file-system check (RESOLVED)

Posted (August 2nd, 2011 at 2:26 pm PST) by patrickm

The shared server ‘bulls’ is currently offline, as it was rebooted and required a file-system integrity check.  This process may take up to a couple hours to complete, but we will post regular updates here!  Please contact Support with any questions!

Update Aug 3rd, 9:11am PDT: The file-system check has completed. The machine has now booted into an operational state and all of the Apache services have started without issue. All affected sites should be back online, but if you’re still experiencing issues please contact our friendly Technical Support staff. -Brandon Previous updates available after the jump →


11:17 am

Shared server ‘bujumbura’ unscheduled downtime (RESOLVED)

Posted (August 2nd, 2011 at 11:17 am PST) by patrickm

The shared server ‘bujumbura’ is currently offline and our Admins are investigating the cause.  We will keep this post updated with as much information as possible!

Update August 2nd, 1:41pm: Turns out our worries were unwarranted!  What we thought was a bad disk drive turned out not to be the case, so the server is back online and running stable since around 11am!  Please contact Support with any questions!


12:07 pm

Shared server ‘johnnyblaze’ down due to RAID card issues — (RESOLVED)

Posted (July 30th, 2011 at 12:07 pm PST) by Oscar

Unfortunately the raid card for shared server johnnyblaze crashed and needs to be replaced.  This should  take about one hour. We sincerely apologize the inconvenience. We will update this post once the raid card has been replaced and the server is back up.

Update July 29th, 5:06pm PDT: Johnnyblaze should be back up and working now! If you’re still having problems with a site hosted on this server, please contact support immediately! –Brian H

Previous updates available after the jump →


11:28 am

Shared server kochab down due to RAID card issues (RESOLVED)

Posted (July 30th, 2011 at 11:28 am PST) by phiya

We’re working to replace the raid card, and restore service as quickly as possible. ETA is within 1 hour. Sorry for the inconvenience. An update will be posted once the issue has been resolved.

Update July 29th, 3:35pm PDT: The server has a new RAID card and is back up and running just fine so this matter is resolved. -John

Previous updates available after the jump →


3:16 pm

VPS host homie-vserver202 downtime for file system check (RESOLVED)

Posted (July 29th, 2011 at 3:16 pm PST) by Oscar

The VPS host server homie-vserver202 is currently offline for file system check. We expect to have this server back online shortly and we will update this post when that happens.

Update July 29th, 5:30pm PDT: The reboot was completed and all VPS machines have been up for the last 20 minutes. If you are still having issues with your VPS or sites being down, please contact our support staff and we will be glad to help you out. -Oscar Previous updates available after the jump →


3:42 am

VPS host homie-vserver138 downtime for unscheduled hardware maintenance

Posted (July 29th, 2011 at 3:42 am PST) by patrickm

The VPS host server homie-vserver138 is currently offline for emergency hardware maintenance.  The RAID card in this server is faulty and we are swapping the drives to a new chassis, so that we do not need to do a full restore.  We expect to have this server back online shortly and we will update this post when that happens.

The following VPSs are affected:

ps19870 ps19898 ps19942 ps20004 ps20053 ps20195 ps20221 ps20483 ps20545 ps20662 ps20798 ps20799 ps21617 ps21731 ps21743 ps21748 ps21785 ps21960 ps22327 ps22595 ps22659 ps22831 ps23071 ps23092 ps23262 ps23325 ps25408 ps25425 ps25429 ps25516 ps25657 ps26375 ps26387 ps26403 ps26506 ps29497 ps29774 ps30295 ps30718 ps34162 ps34264 ps34600 ps35129 ps36441 ps37279 ps37759 ps38190 ps48103 ps48789 ps48794 ps49149 ps49201 ps49445 ps49778 ps55069

Update July 29th, 6:04am PDT: This maintenance is now complete! The new server has been online for about 2 hours now without issue.  Please contact Support with any questions!


4:43 pm

Web hosting machine “grus” still having problems (Resolved)

Posted (July 28th, 2011 at 4:43 pm PST) by Brian@dreamhost

We posted a status update about the web server “grus” being resolved, but it looks like that was a bit premature. We’re opening a new one here, as the server continues to have issues. We’ll continue to update as we get information on its health!

Update July 29th, 12:23am PDT: Our admins have replaced the RAID card on the machine and the service is back up and has been running well with a normal load on the machine. This issue seems to be resolved, if you’re still having trouble please contact support. – JJ G Previous updates available after the jump →


10:20 am

Shared server grus down for RAID card replacement (RESOLVED)

Posted (July 28th, 2011 at 10:20 am PST) by mir

Our admin team is working on replacing the RAID card in this server and expects to have it back up in the next hour. Further updates to come as we have more information. -John

Update July 28th, 2:35am PDT: The server is back up and running but the admin team will thoroughly test it and keep a close eye on it until we are certain it is resolved. -John

Previous updates available after the jump →


11:42 am

.ES domain registrar transfers are failing (RESOLVED)

Posted (July 27th, 2011 at 11:42 am PST) by brians

Again, this only applies to domain registrations that have an .ES extension. We have contacted our .ES affiliate regarding this matter. At this time we are still waiting to receive correspondence from them. Unfortunately, we do not have an ETA at the moment. We sincerely apologize for any inconvenience this has caused and appreciate your continued patience. Please check back for future updates. – Brian S

UPDATE 2011-08-02 11:33AM PST: We have heard from our development team and our VP of Hosting, that this transfer issue is resolved. .ES registrations should be working normally again. If you still see issues, please contact support. – Justin

Previous updates available after the jump →


10:21 am

Power strip problem causing outages on some servers (RESOLVED)

Posted (July 26th, 2011 at 10:21 am PST) by danwy

We’re currently having issues with a power strip in our LAX datacenter. It may be resulting in downtime for the following servers:

Shared web servers:

callisto amalthea chiron theia megrez mekbuda maia markab merak merope menkab menkent

VPS servers

ps47691 ps48920 ps48933 ps48945 ps48946 ps48948 ps48952 ps48953 ps48957 ps48959 ps48967 ps48977 ps48978 ps48983 ps48985 ps49247 ps49271 ps49282 ps49335 ps49360 ps49365 ps49396 ps49422 ps49439 ps49533 ps49561 ps49583 ps49628 ps49682 ps49686 ps49821 ps49951 ps50006 ps50176 ps50316 ps50671 ps50713 ps52478 ps53470 ps53473 ps53975 ps54183 ps54648 ps55239 ps55438 ps55443 ps55446 ps55473 ps55553 ps55572 ps55575 ps55772 ps55790 ps55846 ps55849 ps56648 ps56804 ps56826 ps56943 ps57192 ps48997 ps49001 ps49003 ps49009 ps49011 ps49012 ps49030 ps49033 ps49036 ps49037 ps49038 ps49053 ps49070 ps49077 ps49084 ps49093 ps49096 ps49097 ps49102 ps49135 ps49141 ps49235 ps49293 ps49297 ps49378 ps49389 ps49395 ps49428 ps49485 ps49552 ps49576 ps49579 ps49610 ps49685 ps49723 ps49748 ps49785 ps49980 ps54040 ps54059 ps54530 ps54531 ps54532 ps55423 ps55685 ps55795 ps56895 ps56904 ps57017 ps57488 ps43979 ps47352 ps49354 ps49449 ps49476 ps49482 ps49505 ps49559 ps49564 ps49666 ps49672 ps49675 ps49700 ps49734 ps49736 ps49773 ps49786 ps49800 ps50037 ps50073 ps50075 ps50148 ps50184 ps50234 ps50242 ps50266 ps50268 ps50323 ps50374 ps50399 ps50401 ps50434 ps50440 ps50468 ps50481 ps50519 ps50543 ps50549 ps50557 ps51615 ps52229 ps52246 ps52289 ps52316 ps52389 ps53947 ps54343 ps54494 ps54549 ps55413 ps55866 ps56772 ps56888 ps56907 ps57670 ps57764 ps47396 ps49447 ps49458 ps49479 ps49497 ps49498 ps49534 ps49551 ps49595 ps49601 ps49631 ps49658 ps49706 ps49718 ps49727 ps49743 ps49774 ps49796 ps49832 ps49853 ps49931 ps49938 ps49966 ps49995 ps50076 ps50088 ps50096 ps50097 ps50099 ps42728 ps49461 ps49603 ps49663 ps49680 ps49705 ps49802 ps49899 ps49936 ps49960 ps49963 ps49965 ps50004 ps50007 ps50024 ps50032 ps50103 ps50111 ps50117 ps50142 ps50327 ps50342 ps50353 ps50357 ps50371 ps50378 ps50483 ps50657 ps50689 ps51342 ps51764 ps51804 ps52005 ps52018 ps52130 ps52146 ps52222 ps52237 ps52508 ps52560 ps52848 ps52932 ps53105 ps53110 ps53342 ps53343 ps53360 ps53379 ps53419 ps53464 ps53927 ps54062 ps54113 ps54353 ps54403 ps54497 ps54501 ps55062 ps55556 ps55594 ps56194 ps49459 ps49608 ps49678 ps49823 ps49961 ps49972 ps49986 ps50003 ps50029 ps50095 ps50244 ps50263 ps50437 ps50441 ps50474 ps50485 ps50553 ps50560 ps50561 ps50573 ps50578 ps50596 ps50600 ps50629 ps50659 ps50683 ps50697 ps50730 ps52151 ps52176 ps52315 ps52318 ps52345 ps52346 ps52421 ps52423 ps52428 ps52442 ps52443 ps52444 ps52499 ps52534 ps52945 ps53158 ps53298 ps53353 ps53357 ps53374 ps53413 ps53469 ps53798 ps53825 ps53932 ps54049 ps54112 ps54153 ps54256 ps54344 ps55060 ps55076 ps56722 ps56892 ps57538 ps57595 ps57624 ps57641 ps57676 ps57975

Our admin team is already at work on the power strip and will be restarting the servers shortly. We’ll update this status post as soon as we have any further information.

Update July 26th, 10:50am PDT: We have confirmed that all affected services have been resolved. If you continue to have issues, please contact support and we will be happy to assist you with any remaining problems. -DanW

Previous updates available after the jump →


 
© 1996-2011, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
8:56 am

Shared server cheleb down, being moved to new hardware (RESOLVED)

Posted (July 26th, 2011 at 8:56 am PST) by mir

The server cheleb is currently unstable and our admin team is investigating to determine the cause and correct it. The current ETA is one hour. We’ll update this post once we have completed the correction or have additional information to supply. -John

 

Update July 26th, 12:22PM PDT: The network and hardware are all set and your sites should be back online! Should you find that you are still experiencing any issues on the server, please contact our Technical Support staff! -Brandon

Previous updates available after the jump →


10:24 pm

Upgrading Vserver host machines to squeeze (RESOLVED)

Posted (July 25th, 2011 at 10:24 pm PST) by patrickm

Over the next week or so, we will be upgrading all of our Vserver hosts to the latest version of Debian (squeeze).  This will be an upgrade for the host servers only.  The guest VPSs will not be upgraded at this time.  These upgrades will happen primarily between 7:00pm PDT and 6:00am PDT for the next 7-10 days, or until we have completed the upgrades for all hosts.

The upgrade process will cause approximately 20-25 minutes of downtime for the guests on each host as the upgrade completes, as a restart will be required.  We will do everything we can to minimize this downtime.

We have already tested our process on a number of servers, so we do not expect any issues.  However, we will update this post with any information about issues that arise from this process.  We will also update when we have completed all the upgrades!

Please contact Support if you have any questions about this, or anything else!

Update Aug 12th, 4:00pm PDT: Upgrades are complete, and went swimmingly! If you notice any issues, please let our support team know! –Brian H

Previous updates available after the jump →


5:49 am

Large DDOS attack causing some network instability (RESOLVED)

Posted (July 23rd, 2011 at 5:49 am PST) by jj

We are currently experiencing a rather large DDOS attack affecting our network. This seems to be causing internal DNS problems but further updates will be made when available.

Update July 23, 6:45am PDT: The related internal network issues should no longer be occurring. The DDOS continues as will updates until this is resolved. – JJ G Previous updates available after the jump →


2:27 pm

VPS host machine homie-vserver221 migration (RESOLVED)

Posted (July 21st, 2011 at 2:27 pm PST) by danwy

The VPS server homie-vserver221 is having some issues with its file system and we are in the process of migrating all VPSs on this host to a new host server. Here is a list of VPS servers that will be affected:

ps45566 ps45592 ps45667 ps45675 ps45732 ps45743 ps45776 ps45796 ps45868 ps46092 ps46140 ps46317 ps46517 ps46568 ps46579 ps46620 ps47067 ps47468 ps47469 ps47541 ps47760 ps47827 ps47831 ps48208 ps48214 ps48487 ps48567 ps48583 ps49057 ps49071 ps49165 ps49308 ps49501 ps49523 ps49585 ps49904 ps50247 ps50303 ps50673 ps51475 ps52486 ps54041 ps54901 ps55470 ps55816 ps55817 ps57357

We don’t have an ETA for this issue to be resolved yet, but we will keep this post updated with new information as it becomes available. We’re sorry for any inconvenience this causes you.

Update July 23, 8:00pm PDT: All 46 VPS machines have now been moved off of this server onto more stable machines. This issue is now resolved!

Previous updates available after the jump →


10:40 am

Scheduled Maintenance for Subset of Shared Servers July 23rd. (RESOLVED)

Posted (July 21st, 2011 at 10:40 am PST) by Brandon

We will be replacing the powerstrip on one of our LAX datacenter racks tomorrow between 9am and 11am PDT. There should be at 15-20 minutes of downtime, if any, and we’ll be doing our best to ensure there is as little as possible.
Here is a list of affected webservers:

 

windhoek kathmandu amsterdam wellington managua melekeok islamabad muscat oslo abuja niamey victoria rangoon nouakchott dakar riyadh belgrade

 

We will update this post when we begin maintenance and will keep it up to date with new information as it becomes available!

UPDATE 2011-07-22 9:02 AM PST: We’ve decide to postpone the scheduled maintenance until 2011-07-23. As always, we will update this post as new information is available. Please check back for future updates. – Brian S

UPDATE 2011-07-24 8:52 AM PST: This upgrade was completed without any major issues. Please contact support if you notice any problems. -DanW


10:21 am

Shared server “accra” offline due to filesystem corruption (RESOLVED)

Posted (July 21st, 2011 at 10:21 am PST) by Brandon

Our admin team is currently re-imaging this webserver. The /home partition is completely unaffected and there is no loss of customer data. We are currently estimating that this process will be complete within the next two hours. We will keep this post updated with new information as it becomes available.

 

Update July 21st, 02:05pm PDT: All Apache services and websites are now back online! If you are still experiencing issues please contact our TechSupport staff. -Brandon Previous updates available after the jump →


8:29 am

Irvine Datacenter Networking Issues (RESOLVED)

Posted (July 20th, 2011 at 8:29 am PST) by Justin@DH

Around 8:10AM we started seeing issues connecting to our irvine data center. Our network admins are on the case and looking into the issue. When we have more info we will update this post. The affected servers are:

MySQL VPS: psmysql56854 psmysql56951 psmysql56980 psmysql57091 psmysql57096 psmysql57127 psmysql57174 psmysql57203 psmysql57204 psmysql57205 psmysql57219 psmysql57236 psmysql57544 psmysql57644 psmysql57649 psmysql57685 psmysql56646 psmysql56906 psmysql56915 psmysql56953 psmysql57101 psmysql57125 psmysql57206 psmysql57207 psmysql57208 psmysql57216 psmysql57343 psmysql57377 psmysql57467 psmysql57476 psmysql57549 psmysql57590 psmysql57622 psmysql57777 psmysql56908 psmysql56926 psmysql56955 psmysql56963 psmysql56964 psmysql57046 psmysql57161 psmysql57230 psmysql57300 psmysql57548 psmysql57598 psmysql57621 psmysql57643 psmysql57671 psmysql57677 psmysql57717 psmysql57765 psmysql57770

Shared MySQL: lute banjo ukulele fiddle

Shared webservers: miram mirfak mizar mothallah muliphein muphrid muscida nair-al-saif naos nashira navi nekkar nembus neshmet nihal nunki nusakan okul peacock phact pollux porrima praecipua procyon propus proxima-centauri pulcherrima rana ras-algethi ras-alhague ras-thaoum rasalas rastaban regor regulus rigel rijl-al-awwa rotanev ruchba rukbat sabik sadachbia sadalbari sadalmelik sadalsuud sadatoni sadira sadr saiph salm sargas sarin sarir sasin sceptrum scheat scheddi schedir segin seginus sham shaula sheliak sheratan sinistra situla skat spica sterope-ii sualocin subra suhail sulafat syrma tabit talitha-australis talitha-borealis tarazet taygeta

Dedicated servers: ds1265 ds1267 ds1268 ds1269 ds1271 ds1272 ds1273 ds1274 ds1278 ds1279 ds1280 ds1281 ds1282 ds1285 ds1286 ds1287 ds1289 ds1290 ds1291 ds1295 ds1302 ds1303 ds1306 ds1314 ds1316 ds1317 ds1319 ds1320 ds1322 ds1323 ds1325 ds1326 ds1327 ds1329 ds1333 ds1334 ds1336 ds1337 ds1338 ds1339 ds1343 ds1344 ds1345 ds1347 ds1348 ds1349 ds1350 ds1351 ds1352 ds1353 ds1354 ds1355 ds1356 ds1357 ds1359 ds1360 ds1361 ds1363 ds1364 ds1366 ds1368 ds1369 ds1370 ds1371 ds1372 ds1374 ds1375 ds1376 ds1377 ds1378 ds1379 ds1380 ds1381 ds1382 ds1383 ds1384 ds1385 ds1386 ds1388 ds1389 ds1390 ds1393 ds1394 ds1396 ds1401 ds1403 ds1404 ds1410 ds1411 ds1412 ds1413 ds1414 ds1415 ds1416 ds1418 ds1419 ds1421 ds1424 ds1425 ds1426 ds1427 ds1428 ds1430 ds1431 ds1432 ds1433 ds1434 ds1436 ds1437 ds1438 ds1439 ds1440 ds1441 ds1442 ds1443 ds1444 ds1445 ds1447 ds1449 ds1450 ds1451 ds1453 ds1454 ds1456 ds1457 ds1459 ds1460 ds1464 ds1465 ds1466 ds1467 ds1468 ds1469 ds1471 ds1472 ds1473 ds1474 ds1475 ds1476 ds1477 ds1479 ds1480 ds1481 ds1482 ds1483 ds1484 ds1485 ds1486 ds1487 ds1488 ds1489 ds1491 ds1492 ds1493 ds1494 ds1495 ds1496 ds1497 ds1498 ds1499 ds1500 ds1501 ds1502 ds1503 ds1504 ds1505 ds1506 ds1507 ds1508 ds1509 ds1510 ds1511 ds1512 ds1513 ds1514 ds1515 ds1516 ds1517 ds1518 ds1519 ds1520 ds1521 ds1522 ds1523 ds1524 ds1525 ds1526 ds1527 ds1528 ds1529 ds1530 ds1531 ds1532 ds1533 ds1534 ds1535 ds1536 ds1537 ds1538 ds1539 ds1540 ds1541 ds1542 ds1543 ds1544 ds1545 ds1546 ds1547 ds1548 ds1549 ds1550 ds1551 ds1552 ds1553 ds1554 ds1555 ds1556 ds1557 ds1558 ds1559 ds1560 ds1561 ds1562 ds1563 ds1564 ds1565 ds1566 ds1567 ds1568 ds1569 ds1570 ds1571 ds1572 ds1573 ds1574 ds1575 ds1576 ds1577 ds1578 ds1579 ds1580 ds1581 ds1582 ds1583 ds1584 ds1585 ds1586 ds1587 ds1588 ds1589 ds1590 ds1591 ds1592 ds1593 ds1594 ds1595 ds1596 ds1597 ds1598 ds1599 ds1600 ds1601 ds1602 ds1603 ds1604 ds1605 ds1606 ds1607 ds1608 ds1609 ds1610 ds1611 ds1612 ds1613 ds1614 ds1615 ds1616 ds1617 ds1618 ds1619 ds1620 ds1621 ds1622 ds1623 ds1624 ds1625 ds1626 ds1627 ds1628 ds1630 ds1631 ds1632 ds1633 ds1634 ds1635 ds1636 ds1637 ds1638 ds1639 ds1640 ds1641 ds1642 ds1643 ds1644 ds1645 ds1646 ds1647 ds1648 ds1649 ds1650 ds1651 ds1652 ds1653 ds1654 ds1655 ds1656 ds1657 ds1658 ds1659 ds1660 ds1661 ds1662 ds1663 ds1664 ds1665 ds1666 ds1667 ds1668 ds1669 ds1670 ds1671 ds1672 ds1673 ds1674 ds1675 ds1676 ds1677 ds1678 ds1679 ds1680 ds1681 ds1682 ds1683 ds1684 ds1685 ds1686 ds1687 ds1688 ds1689 ds1690 ds1691 ds1692 ds1693 ds1694 ds1695 ds1696 ds1697

Update July 20th, 12:00pm PDT: While we don’t have all the details yet, it looks like the network connection issues may have only affected internal connections. There was an issue with our SSH tunnel between datacenters that caused issues with our connectivity. Extensive testing from outside of our network returned normal results to all servers listed above. Our network monitoring system is also reporting everything back to normal, so we’re going to go ahead and resolve this post. If you continue to have issues, please contact support to let them know.

Previous updates available after the jump →


4:00 am

Network issues causing downtime (RESOLVED)

Posted (July 20th, 2011 at 4:00 am PST) by patrickm

We are currently experiencing a networking issues which is causing intermittent downtime for Shared, VPS and MySQL hosting.  We are investigating this issue and expect to have it resolved shortly!  We will update this post as soon as possible!

Update July 20th, 4:40am PDT: This network outage lasted only a few minutes and caused minimal downtime for hosting. Please contact Support if you are seeing any issues with your sites!


3:05 pm

Issues with creating tickets in the panel (RESOLVED)

Posted (July 19th, 2011 at 3:05 pm PST) by Oscar

We have noticed that customers are having issues creating tickets within the panel. Our development team is working on this and will get this fixed as soon as possible.  If you need to contact support, please use our “Live Chat” feature  from your panel if you have it enabled or you can reach us through our contact link here:

http://www.dreamhost.com/contact.cgi?regarding=sales

We will be updating this post as soon as we have more information.

Update July 19th, 3:30pm PDT: This is completely fixed and you can now submit tickets from the panel! -Oscar


2:13 am

VPS host homie-vserver221 downtime for unscheduled file-system check (RESOLVED)

Posted (July 19th, 2011 at 2:13 am PST) by patrickm

The VPS host server homie-vserver221 was rebooted and came back with a read-only file-system again.  Due to the repeat issues with this, a file-system check is now taking place on the server.  The server will remain offline during this check and we will post updates as this progresses.

The following VPS guests are affected by this downtime:

ps45566 ps45592 ps45667 ps45675 ps45732 ps45743 ps45776 ps45796 ps45868 ps46092 ps46140 ps46317 ps46517 ps46568 ps46579 ps46620 ps47067 ps47468 ps47469 ps47541 ps47760 ps47827 ps47831 ps48208 ps48214 ps48487 ps48567 ps48583 ps49057 ps49071 ps49165 ps49308 ps49501 ps49523 ps49585 ps49904 ps50247 ps50303 ps50673 ps51475 ps52486 ps54041 ps54901 ps55470 ps55816 ps55817 ps57357

Update July 19th, 7:30am PDT: The file system check is still running, our admin team reporting that it is still on pass 1. We will update this post again soon with the status. – JustinK

Update July 19th, 11:30am PDT: The file system check has finished, and the vps should be all running and back to normal. If you are still experiencing issues with your vps, please contact support. – JustinK


 
© 1996-2011, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
9:22 pm

Shared server “fomalhaut” down due to bad RAID card (RESOLVED)

Posted (July 18th, 2011 at 9:22 pm PST) by Jason C

Customers on “fomalhaut” may notice issues with their sites and server being available. After investigating an matter involving server stability, our admin team has decided to dispatch someone to the datacenter to replace the RAID card. Hopefully the outage should be brief, but we’ll do our best to keep you updated here as things progress.

Update July 19th, 1:45am PDT: We were able to replace the RAID card for this server and it is now back online! Please contact Support with any questions! -PatrickM


6:49 am

Shared server namor unscheduled file-system check (RESOLVED)

Posted (July 16th, 2011 at 6:49 am PST) by jj

The shared server namor was having raid issues this morning. The raid card has been replaced and after a reboot the sever is now performing a necessary file system check. Our admins are monitoring the machine and we are doing what we can to speed along the process. Updates will come as they are made available.

Update July 16th, 8:30am PST: The server is still on the first pass of the fsck. The estimated time of completion is in about 3-4 hours from now. We’ll update this post when we have more to share. — John

Update July 16th, 11:00am PST: The fsck is completely done now and the server is back up. All of your sites should be now. If you have any further issues with your sites being down, please contact our support team and we will be glad to look further into for you. — Oscar


12:41 am

Shared server matchless restoring from backups (RESOLVED)

Posted (July 16th, 2011 at 12:41 am PST) by jj

We have experienced a failure of two drives on the shared server matchless. The drives that failed are a matched pair which caused the entire RAID array to go down. The drives are being replaced and are being restored from backups. The server will remain online during this process. More information will be posted as it becomes available.

Update July 22nd, 9:26pm PDT: The restore has finished!  If you notice any trouble with your websites or have any questions, please contact our friendly support staff! –JasonY

Previous updates available after the jump →


4:30 pm

Shared server matchless downtime due to file system corruption (RESOLVED)

Posted (July 15th, 2011 at 4:30 pm PST) by mir

Our admin team is actively working on this issue and we are hoping to see it back up in the next hour or so if the fsck takes care of it. We’ll update this post once we have more information for you. -John

Update July 15th, 9:40pm PDT: Ok, file system check is done, and services should be restored! If you’re still seeing problems, please let support know asap! — Brian H

Previous updates available after the jump →


4:27 pm

Shared server bishop downtime due to file system corruption (RESOLVED)

Posted (July 15th, 2011 at 4:27 pm PST) by mir

Our admin team is actively working on this issue and we are hoping to see it back up in the next hour or so if the fsck takes care of it. We’ll update this post once we have more information for you. -John

Update July 15th, 4:30pm PDT: I apologize, the server name reported was incorrect, bishop is just fine (matchless is the server requiring an fsck). -John


3:12 am

Shared server ‘paulreedsmith’ unscheduled downtime for RAID controller maintenance (RESOLVED)

Posted (July 15th, 2011 at 3:12 am PST) by patrickm

The shared web server ‘paulreedsmith’ is currently experiencing issues with its RAID controller.  We are taking down the server to replace the RAID controller.  We will update this post when this is complete!

Update July 15th, 5:29am PDT: Apologies for the lack of follow-up to this post! The RAID controller was swapped out shortly after this post was made and the server has been back up and running with no problems since that time! Please contact Support if you are on this server and you are having any issues, or if you simply have questions!


7:59 am

Shared server bishop downtime due to disk corruption (RESOLVED)

Posted (July 14th, 2011 at 7:59 am PST) by mir

We’ve dispatched an admin to work on this issue and are hoping to see it back up in the next half hour or so if the fsck takes care of it. We’ll update this post once we have more information for you. -John

Update July 14th, 9:00 am PDT: The fsck has completed successfully and services are coming back online now for this server. Everything should be fully functional in the next 10 minutes or so. -John

Previous updates available after the jump →


4:42 am

Shared server mahasim downtime due to RAID issues (RESOLVED)

Posted (July 14th, 2011 at 4:42 am PST) by patrickm

We are currently investigating degraded performance on the shared server mahasim.  We are seeing issues with the RAID array, which is likely causing these problems.  We will update this post with more info as it becomes available!

Update July 14th, 7:50 am PDT: Everything is back up and running with ten times the disk performance it was previously capable of with the bad disk. This issue is now resolved. -John

Previous updates available after the jump →


11:53 am

WordPress 3.2.1 now available in one-click installer

Posted (July 13th, 2011 at 11:53 am PST) by Jason C

Customers who have used our one-click installer should be aware that WordPress 3.2.1 is now available. This has a handful of fixes for some small but annoying bugs in last week’s WordPress 3.2 release.

Our auto upgrade function is running for the customers who have opted-in to that and you should get a message about the upgrade being applied to your install in the next few days. If you can’t wait though, you can upgrade via your WordPress admin area (Dashboard > Upgrades) without any worries.


7:24 am

VPS Host Machine homie-vserver08 – Unscheduled file-system check (RESOLVED)

Posted (July 12th, 2011 at 7:24 am PST) by Justin@DH

The webserver VPS host machine homie-vserver08 was having raid issues this morning, and after a reboot is doing a necessary file system check. Our admin is monitoring it and we are doing all we can to speed up the process. The affected VPS webservers are:

ps10167 ps10940 ps11997 ps11998 ps12218 ps12316 ps13132 ps13163 ps13166 ps13934 ps14021 ps14022 ps14387 ps14447 ps17178 ps17965 ps17970 ps19067 ps27908 ps27927 ps28163 ps29410 ps30790 ps33741 ps34292 ps35246 ps36180 ps38909 ps43664 ps43838 ps43853 ps43906 ps44311 ps44378 ps44381 ps4481 ps4494 ps4498 ps4501 ps4512 ps4515 ps4531 ps4534 ps4541 ps4543 ps4700 ps4703 ps4737 ps4826 ps4832 ps4994 ps50533 ps53510 ps54141 ps54163 ps54194 ps54330 ps55113 ps55117 ps55130 ps55876 ps55877 ps55902 ps8187 ps8441 ps9498

Update 2011-07-12 9:00am – The file system check has completed, and this vserver is back up and running. If you still see any related issues to this server, please contact support.


 
© 1996-2011, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
2:17 pm

Shared machine “pisces” was suffering from memory issues (RESOLVED)

Posted (July 11th, 2011 at 2:17 pm PST) by Jason C

Customers on the server “pisces” may notice that their sites are current inaccessible. As it stands, the server is down due to issues with the system memory. We’re dispatching an admin to the datacenter now and hopefully we’ll be able to resolve this quickly.  More information will be added to this post as this matter progresses.

Update 4:42pm PDT: The RAM upgrade is complete and things are running smoothly now. If you happen to see any further issues with pisces, please don’t hesitate to contact our support team. – Jason

Previous updates available after the jump →


7:23 pm

VPS host server homie-vserver07 crashed and is being rebooted! (RESOLVED)

Posted (July 8th, 2011 at 7:23 pm PST) by Brian@dreamhost

Unfortunately, the VPS parent host, homie-vserver07 needed to be rebooted. In doing this, the hard drives on the machine need to be checked by the operating system. That is currently happening now, and should be finished shortly. We’ll update this post with more info as it’s taken care of. Sorry about the problems this causes! The following VPS machines are affected by this issue:

ps11283 ps11550 ps11944 ps11996 ps12637 ps12968 ps13010 ps13076 ps18573 ps19523 ps20112 ps21009 ps22766 ps28537 ps36030 ps38105 ps38163 ps43919 ps43922 ps43926 ps43927 ps4431 ps4434 ps4446 ps4449 ps4451 ps4453 ps4455 ps4458 ps4459 ps4674 ps4812 ps4828 ps4856 ps4864 ps4922 ps4933 ps5015 ps5047 ps5052 ps5065 ps5067 ps5096 ps5101 ps5108 ps6784 ps8430 ps8443

Update: July 8th, 8:50pm PDT: Homie-vserver07 is now up and running. All guests have been started and are running as well. If you see any problems, please let us know ASAP! -Brian H

Previous updates available after the jump →


8:45 am

Shared machine Laney is having its RAID card replaced (RESOLVED)

Posted (July 7th, 2011 at 8:45 am PST) by brians

The shared machine laney has been experiencing stability issues the past few hours. The source of the problem appears to be a faulty raid card. Our admins are in the process of replacing the raid card at the moment. Downtime should not last more than 10 – 15 minutes. Sorry for any inconvenience this has caused. Your continued patience is greatly appreciated. We will update this post once the machine is back online. Please check back for future updates.

UPDATE 2011-07-07 8:50AM PST:

The machine is back online. All services are up and running. Again, we sincerely apologize for any inconvenience the downtime may have caused. If you are still encountering problems, please contact support. – Brian S


2:38 pm

DreamBook data being moved! (RESOLVED)

Posted (July 6th, 2011 at 2:38 pm PST) by Brian@dreamhost

We’re currently moving the data stored on our guestbook application, http://dreambook.com to a different file server. You shouldn’t notice any downtime related to this, but we thought we’d let you know just in case! If you do see problems, please let us know at support =)

Update July 6th, 4:20pm PDT: This should now be resolved! Again, if you notice any problems, please let us know.


9:46 am

Shared server “dschubba” experiencing networking problems due to UDP flood. (RESOLVED)

Posted (July 6th, 2011 at 9:46 am PST) by Brandon

Customers with sites on the shared server “dschubba” may be experiencing site downtime or other networking issues due to a UDP flood causing networking issues. We’ve isolated the Apache service being targeted and will have things resolved as quickly as possible.

Update July 6th, 10:00am PDT: The server is now looking improved. The target Apache service, “apache2-cabo”, was renumbered, so customers with sites on this Apache service will need to wait out any DNS updates. -Brandon


8:23 am

Washingtondc Having Networking Issues Causing Downtime (RESOLVED)

Posted (July 5th, 2011 at 8:23 am PST) by Justin@DH

Our admins are currently looking into networking issues on the washingtondc server, which is causing sites on it downtime. They are investigating if it is a networking hardware issue, or perhaps a DDoS. Once we have more information, or the issue is fixed, we will update this post.

Update July 5th, 09:30am PDT: The server is back to normal but a handful of the domains on the specific apache which was hosting the targeted domain will be unavailable as DNS updates (this should complete within the next hour or so). -John

Previous updates available after the jump →


9:29 pm

Shared MySQL server “Smelt” back up and running after RAID card replacement (RESOLVED)

Posted (June 30th, 2011 at 9:29 pm PST) by Jason C

Customers on the shared MySQL server “smelt” will be noticing issues getting thru to their databases. Our admin team spotted an issue with the server a short time ago and they are now in the process of replacing the RAID card.

We will keep this post updated with developments as this issue progresses.

Update July 1st, 12:00pm PDT: All databases on the “smelt” MySQL server should now be up and running, including those on the “dearmon” service. Should you still be experiencing issues, please let our Technical Support staff know! -Brandon

Previous updates available after the jump →


5:48 pm

Issues registering me.uk domains (RESOLVED)

Posted (June 30th, 2011 at 5:48 pm PST) by Oscar

We are currently having issues processing domain registrations for me.uk domains. We are looking into the issue will update this post as soon as it’s resolved.

Update June 30th, 6:10pm PST: This is resolved now and any domain registration for .me.uk will process without any issues now. –Oscar


3:39 am

Shared server oslo having RAID issues causing downtime (RESOLVED)

Posted (June 30th, 2011 at 3:39 am PST) by patrickm

The shared server ‘oslo’ is down at this time due to issues with the RAID card.  We are working to replace this card as quickly as possible and we will update this post with that status!

Update June 30th, 4:41am Pacific: The RAID card for this server has been replaced and the server is now back online! Please contact Support with any questions!


8:53 am

VPS Host Machine homie-vserver118 – Scheduled File System Check (RESOLVED)

Posted (June 29th, 2011 at 8:53 am PST) by Justin@DH

The webserver homie-vserver118, which hosts the following webserver VPS:

ps17627 ps17699 ps17743 ps17787 ps17859 ps18123 ps18128 ps18129 ps18137 ps18143 ps18220 ps18221 ps18227 ps18228 ps18229 ps18235 ps18253 ps18262 ps18263 ps18264 ps18328 ps19277 ps19443 ps22810 ps22910 ps22911 ps22913 ps23136 ps23137 ps26609 ps26873 ps26909 ps26962 ps26963 ps26964 ps26975 ps27413 ps27758 ps28516 ps28520 ps28996 ps29197 ps29578 ps29950 ps35369 ps37216 ps37371 ps37729 ps37730 ps38096 ps38171 ps38672 ps39436 ps41146 ps41746 ps42010 ps42809 ps44874 ps44886 ps46511 ps46697 ps47513 ps47689 ps47710 ps48254 ps48267 ps48270 ps48749 ps48823 ps49330 ps49862 ps52390

has been having repeat issues with its RAID, where it would go read-only. The admins wish to do a file system check on it, as preventative maintenance before more serious RAID issues occur. The file system check will require the VPS to be turned shut down, and will begin at midnight pacific time. We will update the status post as it progresses to give a rough ETA and % done update. If you have any questions, please contact support.

Update June 30th, 7:00am Pacific: We have heard from our admins and noc team, that the file system check has completed and the vservers are starting up. It may take a few minutes for this to complete, otherwise the maintenance has completed. If you still see issues after the file system check, please contact support.

Previous updates available after the jump →


 
© 1996-2011, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
6:10 pm

Shared server “sainthelier” down for unscheduled file system check (RESOLVED)

Posted (June 27th, 2011 at 6:10 pm PST) by Jason C

Our admins have let me know that “sainthelier” had to be brought down for an unscheduled check of its file system. This means users on the server will not be able to see their sites for a short while. We will update this post with more information as the check progresses, so stay tuned!

Update June 28th, 2:00am PDT: The file-system check on this server is now complete and the server is back online! Please contact Support with any issues! -PatrickM

Previous updates available after the jump →


4:33 pm

Shared server “bujumbura” was down for unscheduled file system check. (RESOLVED)

Posted (June 27th, 2011 at 4:33 pm PST) by Jason C

While troubleshooting some issues on bujumbura, our admin team was forced to take down the server for a file system check. Hopefully this isn’t an indication of a bigger issue, but if it is you can be sure that we’ll let you know here. Keep an eye on this post for further updates!

Update 5:29pm PDT: After having to run a quick restore on one of the partitions, the admin team has given us the all clear. If you continue to see issues with your sites on bujumbura, please don’t hesitate to contact our support team!


2:36 pm

The cebalrai server is having RAID issues (RESOLVED)

Posted (June 26th, 2011 at 2:36 pm PST) by danwy

The server ‘cebalrai’ is currently having issues with its RAID array and has become unresponsive. Our admins are currently on their way to physically check on the machine in the datacenter and get it back up and running. We don’t have an ETA on when this will be resolved yet, but we will update this post regularly with more information. We’re sorry for any inconvenience this causes you.

Update 3:00pm PDT: – Our admins were able to get things restarted and the server is back up and running now. If you have any further problems on this server, please contact support to let them know.


6:59 am

Shared server namor down for unscheduled file-system check (RESOLVED)

Posted (June 26th, 2011 at 6:59 am PST) by Justin@DH

We apologize for the downtime, but the shared server namor is down for a file-system integrity check due to the server’s file-system switching to read-only. The server has a somewhat large amount of data, 3.2TB, so unfortunately the file system check may take up to 6+ hours. We will update this post regularly with progress updates, if you are affected please check back for more information.

Update 8:00am PDT: – The file system check is continuing normally, it is on pass 1.
Update 9:00am PDT: – The file system check is still going, still on pass 1.
Update 10:00am PDT: – No new info, fsck is still running, pass 1.
Update 11:00am PDT: – The file system check finished faster than expected, the server is back up and services restored.


9:37 pm

Support attachment submission offline, Sunday 10pm PDT, June 26th, 2011

Posted (June 24th, 2011 at 9:37 pm PST) by Brian@dreamhost

We’re making a small change to our support system on Sunday, June 26th at 10pm PDT. This change will take down attachment submission when sending in a support request. This function of our support system should only be offline for about 10-20 minutes. We apologize for the inconvenience this causes!


3:23 pm

VPS host machine “Homie-vserver180″ was offline for RAID replacement (RESOLVED)

Posted (June 24th, 2011 at 3:23 pm PST) by Brandon

Recent instability on the VPS host machine homie-vserver180 has been tracked to a faulty RAID card. It’ll be going offline for a replacement now. Typically this process should take less than 30 minutes, but we’ll keep this up to date with any pertinent information.

List of affected VPS guests:

 

ps27055 ps27126 ps27128 ps27129 ps27130 ps27131 ps27132 ps27137 ps27138 ps27139 ps27140 ps27141 ps27142 ps27144 ps27145 ps27146 ps27147 ps27149 ps27150 ps27152 ps27153 ps27181 ps27185 ps27230 ps27231 ps27232 ps27234 ps27235 ps27237 ps27239 ps27712 ps42331 ps42598 ps47407 ps47790 ps47954 ps48544 ps48546 ps48552 ps54111 ps54554 ps54559 ps54560 ps54574 ps54657

Update June 24th, 3:30pm PST: The RAID card replacement has been finished without issue! If you’re experiencing any issues please contact our Technical Support staff!


5:03 pm

Some FilesForever files lost

Posted (June 23rd, 2011 at 5:03 pm PST) by Oscar

Unfortunately  we have some lost some files hosted within the file forever server. We have tried to recover as many files as possible however, some have been permanently lost. If you are still missing files on your account, please contact our support team by submitting a support ticket. We will provide refunds for any files that were paid for and lost.

CLARIFICATION: Sorry for any confusion this post may have caused. The problem that caused the data loss occurred in early April. We have not had any problems since that time. Our sincere apologies to those affected.


4:45 am

Shared server mensa down for RAID card replacement (RESOLVED)

Posted (June 23rd, 2011 at 4:45 am PST) by patrickm

We are bringing down the shared server ‘mensa’ for a RAID card replacement.  This should only take a few minutes and we will update this post as soon as the work is completed!  Thanks for your patience and understanding!

Update June 23rd, 5:52am PST: This is now complete!  The server is back up and running with the new RAID card.  Please contact Support if you have any issues with your sites!


6:17 pm

.tv domain registration transfers currently having issues. (RESOLVED)

Posted (June 22nd, 2011 at 6:17 pm PST) by Oscar

.tv domain registration transfers are currently failing, however this is not affecting .tv registrations. We are currently working on the issue and will update this post as soon as it’s fixed.

Update June 22 8:25pm PST: This issue has now been resolved and .tv domain registration transfers are working now. -Oscar


12:02 pm

Sites down on shared server Capone (RESOLVED)

Posted (June 22nd, 2011 at 12:02 pm PST) by Brandon

Apache services on the shared webserver “Capone” are currently offline due to the /home partition having become full. Our administrative team is currently resolving this matter, and hopes to have things fixed up within the hour. We’ll keep this post updated with information as it becomes available. Thank you for your patience.

 

Update June 22 12:25am PST: This issue has now been resolved and all sites should be back online. Should you still be experiencing issues stemming from this, please contact our Technical Support staff.


 
© 1996-2011, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
1:07 am

Shared server mensa down (RESOLVED)

Posted (June 22nd, 2011 at 1:07 am PST) by patrickm

We are investigating a possible issue with the RAID card on the shared server mensa.  The server is down at this time, but we are working on getting it back up and running as quickly as possible.  We will update this post with more details shortly!

Update June 22 11:25am PST: We’ve isolated the problem and have now rectified it. Your sites should be back online, but things may be a little slow for some time. Please feel free to contact our Technical Support representatives should any issues be ongoing. -Brandon

Previous updates available after the jump →


11:42 pm

Shared server namor down (RESOLVED)

Posted (June 21st, 2011 at 11:42 pm PST) by patrickm

The shared server namor is currently down.  Our engineers are working on the issue and we hope to have this server back online shortly!  We will update this post with additional information as it becomes available!

Update June 21st 3:45am PST: We found that the issue with this server was the RAID card, which has now been replaced.  The server is back up and running, but please contact Support if you notice any issues!


4:44 pm

Shared server “aldhibah” has become unresponsive. (RESOLVED)

Posted (June 21st, 2011 at 4:44 pm PST) by Jason C

Our admin team has noticed an issue with the shared server “aldhibah” becoming unresponsive. They are currently looking into why exactly it has done so, but it may take a little longer than normal for it to get back up and be stable. Please keep an eye here for further updates.

Update 5:15pm PST: Our admin team has let us know that the server was hung at a command prompt and needed a little TLC. Everything is back to normal as of now. We are rather sorry for any issues this has caused. If you happen to see any more problems, don’t hesitate to let us know! – Jason


11:33 am

E-mail delayed for some servers (RESOLVED)

Posted (June 21st, 2011 at 11:33 am PST) by danwy

We’re currently experiencing some mail delay issues on our master mail cluster (homiemail-master). We’re not quite sure of what the cause is yet, but our admin team is aware of the issue and working on getting it resolved ASAP. We believe the problem is related with the MX Records of the servers, but don’t have any additional details at this time. We will update this post as soon as we have more information.

Update 4:30PM PST: E-mail is now being delivered in a timely manner. The mail queues are looking much improved and this issue should now be resolved.  The roof cause of the delay was a MySQL failover. This has been repaired and all queued mail began delivering normally. There was no mail lost. If you are still experiencing any issues stemming from this please contact our Technical Support staff. -Brandon

Previous updates available after the jump →


6:11 pm

MySQL server “megalodon” was running with degraded performance. (RESOLVED)

Posted (June 20th, 2011 at 6:11 pm PST) by Jason C

Our admin team is currently investigating what appears to be a misbehaving user on the shared MySQL server “megalodon”. As of this moment, our customers on that server are seeing some timeouts and degraded performance for databases on that server. We are doing everything we can to get this sorted out as quickly as possible, so please keep an eye on this post for any and all updates related to this matter.

Update 6:38pm PST: The user has been dealt with and the admin on duty has gone ahead and given us the all clear on this matter. If you are on “megalodon” and continue to see issues, please don’t hesitate to contact our support team.


9:49 am

Professionally signed SSL certificate orders/renewals not working. (RESOLVED)

Posted (June 20th, 2011 at 9:49 am PST) by David

Unfortunately our control panel is having a problem initiating SSL certificate orders and renewals at the moment. It is not displaying the available Domain Control Validation e-mail addresses when the order is initiated and will finally give an error. Our developers are aware of the problem and are working to resolve it as quickly as possible. Please check back here for updates.

We apologize for any inconvenience this may cause.

UPDATE: 10:15am PST
The control panel is working again now, we’re just waiting to confirm once we’ve seen an order successfully processed before declaring this problem resolved.

UPDATE: 12:08am PST
It’s been verified that the control panel is working properly and I’ve seen at least one SSL cert order has been completed successfully already so this issue can be closed. If you have any problems with secure hosting or the ordering/updating of SSL certificates in our system please contact technical support.

We apologize for any inconvenience this problem may have caused.


5:12 am

Network issues causing downtime for some sites and services *Resolved*

Posted (June 20th, 2011 at 5:12 am PST) by patrickm

We are currently experiencing a network issue between two of our data centers, which is causing downtime for some hosted domains and services.  We are investigating this issue now and will update this post when we have more details!

Update June 20th, 6:25AM Pacific: – Our network engineers have made the changes necessary to stabilize the connection between the data centers, and normal service should be restored at this time. The cause is still being investigated, and if any further changes are necessary that require a maintenance period, a new post will be made with those details. Our apologies for the issues this morning caused by the networking issue, we are continuing to monitor the situation.


12:19 pm

Shared server “guild” down due RAID card replacement (RESOLVED)

Posted (June 18th, 2011 at 12:19 pm PST) by Oscar

Our sincere apologies for the downtime. The shared server guild is having issues with its raid card and our admin team is currently replacing it. We will update this post as soon as the card has been replaced and the server is back up.

Update June 18th 2011, 12:36pm: The raid card has been replaced and the server is now back up. Your sites are now back up and resolving as well. If you are still having issues with your sites being down, please contact our support staff and we will be glad to look into for you. – Oscar


12:09 pm

Shared server “dalitz” down for emergency maintenance (RESOLVED)

Posted (June 18th, 2011 at 12:09 pm PST) by Oscar

We apologize for the inconvenience. The shared server “dalitz” is having hardware issues at this time. Our admin team will be replacing the hardware on the machine. In essence we are mounting all drives to a new chassis. We will update this post as soon as we have more information.

Update June 18th 2011, 1:30pm: The chassis swap was not needed and we only needed to replace the power supply unit. This is all done now and the server is back up. If you have any further issues with your sites not loading on “dalitz”, please contact our support team and we will be glad to look into it for you.. – Oscar


6:40 am

Shared server esp down for unscheduled file-system check (RESOLVED)

Posted (June 18th, 2011 at 6:40 am PST) by robert

We apologize for the downtime, but the shared server esp is down for a file-system integrity check due to the server’s file-system switching to read-only.  Please keep an eye on this post for any and all future updates.

Update June 18th 2011, 8:46am: The file-system check is approximately 50% done now!  We will update this post shortly! – Oscar

Update June 18th 2011, 12:13pm: The file-system check is going. This is taking longer than expected, but we will update this post as soon as the file system check progresses. – Oscar

Update June 18th 2011, 1:33pm: The file-system check is now complete and the server is back up. If you are still having issues with your sites being down, please contact our support staff and we will be glad to look into for you. -Oscar


 
© 1996-2011, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
12:30 am

Shared server sainthelier downtime for unscheduled file-system check (RESOLVED)

Posted (June 17th, 2011 at 12:30 am PST) by patrickm

We apologize for the downtime, but the shared server sainthelier is down for a file-system integrity check due to the server’s file-system switching to read-only.  We will update this post with further details very soon!

Update June 17th 2011, 2:12am: The file-system check is approximately 50% done now!  We will update this post shortly!

Update June 17th 2011, 5:50am: This fschk is now complete and the server is back online!  If you notice any issues with your hosting from this server, please contact Support!


4:44 pm

Shared server “earth” back after unscheduled file system check. (RESOLVED)

Posted (June 16th, 2011 at 4:44 pm PST) by Jason C

After hanging post reboot, the shared server “earth” is in the process of performing a file system check. Our admin team is currently doing what they can to make sure this process goes smoothly. Please keep an eye on this post for any and all future updates.

Update 5:32pm PST: The bad disk has been taken out of the equation and the server is back up and plugging along without issues. If you are on “earth” and notice anything out of the ordinary, please contact our support staff ASAP.

Previous updates available after the jump →


12:06 pm

1 hour mail server maintenance window scheduled for 10pm PDT on June 16th, 2011 (RESOLVED)

Posted (June 15th, 2011 at 12:06 pm PST) by Brian@dreamhost

We’ll be doing some maintenance on two file servers, “staythirsty” and “myfriends” on June 16th, 2011 at 10pm PDT. You can see if you’re on these particular file servers by logging into our web panel, and clicking “account status” on the upper right hand side of the page. They should be shown in the “file server” area towards the middle of the menu that expands out when clicked.

We anticipate this maintenance window lasting about an hour and will update this post as things happen. If your mail is not hosted on one of these file servers, and you’re having problems, please let support know right away.

Update June 16th 2011, 10:02pm PST: This is happening now, we expect downtime of these email addresses of about an hour. –Brian H

Update June 16th 2011, 11:21am PST: This maintenance is now complete and all services are back online!  –PatrickM


4:19 am

Vserver host homie-vserver160 downtime for RAID card replacement (RESOLVED)

Posted (June 15th, 2011 at 4:19 am PST) by patrickm

We apologize for this downtime, as a quick reboot to replace the RAID card on this server has caused more issues and the server is still down. We are working to get this server up as quickly as possible and will update this post with any new details.

Affected VPS guests:

ps23347 ps23389 ps23423 ps23432 ps23462 ps23635 ps23669 ps23778 ps24163 ps24210 ps24295 ps24415 ps24490 ps24509 ps24530 ps24558 ps24794 ps24914 ps24921 ps24927 ps24976 ps25021 ps25028 ps25300 ps25302 ps25357 ps25380 ps25401 ps25431 ps25879 ps26543 ps26659 ps26774 ps26795 ps27767 ps27821 ps28063 ps28886 ps28964 ps29073 ps29181 ps29267 ps35496 ps35894 ps36271 ps36645 ps37992 ps39870 ps41116 ps41955 ps41956

Update June 15th 5:09am PST: This maintenance is now complete! All VPS guests are back up and running.


3:17 am

Shared server danelectro back online after file system check (RESOLVED)

Posted (June 13th, 2011 at 3:17 am PST) by patrickm

The shared web server danelectro was rebooted and came back with a read-only file-system, which necessitated a file-system integrity check.  The server will continue to be down while this check is completed. We will update this post with the progress of the fschk and any other details!

Update June 13th 3:14pm PST: Sites on danelectro are up and running without issue. Should you still be experiencing issues please contact our support staff! -Brandon

Previous updates available after the jump →


6:12 pm

VPS host machine homie-vserver41 downtime while restoring from backups (RESOLVED)

Posted (June 12th, 2011 at 6:12 pm PST) by brettg

Our VPS host server homie-vserver41 had to be forcefully rebooted and did not pass a fsck (filesystem check) on it’s primary partition. The server is now running on new hardware, but will remain down while we sync all data from our backup server to the new drives.

This is currently affecting the following servers:

ps10064 ps10066 ps10105 ps10300 ps10305 ps10307 ps10377 ps10609 ps12001 ps25269 ps25426 ps26140 ps26308 ps28525 ps29110 ps29127 ps29360 ps29878 ps30275 ps30301 ps34012 ps34840 ps34977 ps37655 ps42072 ps43572 ps44747 ps44836 ps49356 ps49584 ps49837 ps50164 ps50551 ps52295 ps52322 ps7618 ps8461 ps8462 ps8463 ps8464 ps8465 ps8466 ps8467 ps8468 ps8469 ps8470 ps8471 ps8472 ps8473 ps8474 ps8475 ps8476 ps8477 ps8478 ps8479 ps8480 ps8481 ps8482 ps8483 ps8484 ps8485 ps8486 ps8487 ps8488 ps8489 ps8490 ps8491 ps8492 ps8493 ps8494 ps8495 ps8496 ps8497 ps8498 ps8499 ps8500 ps8501 ps8502 ps8503 ps8504 ps8505 ps8506 ps8507 ps8508 ps8509 ps8510 ps8511 ps8512 ps8513 ps8514 ps8515 ps8516 ps8517 ps8518 ps8519 ps8520 ps8521 ps8522 ps8523 ps8524 ps8525 ps8526 ps8527 ps8528 ps8529 ps8530 ps8531 ps8532 ps8533 ps8534 ps8535 ps8536 ps8537 ps8538 ps8539 ps8540 ps8541 ps8542 ps8543 ps8544 ps8545 ps8546 ps8547 ps8548 ps8549 ps8550 ps8551 ps8552 ps8553 ps8554 ps8555 ps8556 ps8557 ps8558 ps8559 ps8560 ps8561 ps8562 ps8563 ps8564 ps8565 ps8566 ps8567 ps8568 ps8569 ps8570 ps8571 ps8572 ps8573 ps8574 ps8575 ps8576 ps8577 ps8578 ps8579 ps8581 ps8582 ps8583 ps8585 ps8587 ps8588 ps8589 ps8591 ps8592 ps8593 ps8594 ps8596 ps8597 ps8598 ps8600 ps8601 ps8602 ps8603 ps8604 ps8606 ps8607 ps8608 ps8609 ps8610 ps8611 ps8612 ps8613 ps8614 ps8615 ps8616 ps8617 ps8618 ps8619 ps8620 ps8621 ps8622 ps8623 ps8624 ps8625 ps8626 ps8627 ps8628 ps8629 ps8630 ps8631 ps8632 ps8633 ps8634 ps8635 ps8636 ps8637 ps8638 ps8639 ps8640 ps8641 ps8642 ps8643 ps8644 ps8645 ps8646 ps8647 ps8648 ps8649 ps8651 ps8652 ps8653 ps8967 ps8968 ps8971 ps8972 ps8975 ps8981 ps8987 ps8994 ps8996 ps9125 ps9127 ps9130 ps9131 ps9133 ps9142 ps9151 ps9318 ps9342 ps9540 ps9574 ps9579 ps9608 ps9831 ps9833 ps9842 ps9853 ps9876

We will continue to regularly update this status post, until the issue is resolved. We apologize for the inconvenience, if you have any questions please contact our support staff via the Web Panel.

Update June 15th 3L40am PST: All VPS guests on this vserver host are now restored and operational again! If you are still having any issues with your VPS, please contact Support and we will be happy to investigate! Thank you for your patience during this restore!

Previous updates available after the jump →


2:51 pm

New TLD domain registration having sporadic problems

Posted (June 11th, 2011 at 2:51 pm PST) by Brian@dreamhost

Currently we’re seeing some sporadic problems when registering one of our fancy new “Top Level Domain” names (or TLD for short). The problem would show up to customers as a delay in being processed. Due to these issues, we’ve disabled registration of “.ca”, “.co.uk”, “.es” and “.eu” domains.

We apologize for this, and are working to get the issue fixed. Normal .com, .net and .org registrations will not be effected by this.

Update June 20th 2:20PM We have resolved the issues with the .ca, .co.uk and .eu domains, but are still having issues getting .es domains back up for registration. It’s going to take some more work on our panel backend to make these registrations function correctly and we don’t currently have an ETA on when it will be resolved but hope it will be in the next couple of days. Once we get this issue resolved, .es domains will be able to be registered via the panel again. -DanW

Previous updates available after the jump →


12:27 pm

.DE registry will be offline on June 15, 2011 (RESOLVED)

Posted (June 10th, 2011 at 12:27 pm PST) by brians

The .DE Registry has informed us that they will be undergoing a planned downtime for maintenance on June 15, 2011. They estimate that the maintenance should only take 30 minutes. But you never know how long the downtime may actually last. If you need to register a .DE domain, it’s advisable that you do it before 2011-06-15. There it is, you have been warned!


10:01 am

Shared machine Galactus back online after downtime due to networking issues. (RESOLVED)

Posted (June 9th, 2011 at 10:01 am PST) by Brandon

Shared server Galactus is currently offline due to networking issues. Our network operations staff are currently working on resolving the issue which may entail a replacement of networking hardware. We will get this back online as soon as possible, and will keep this post up to date with the most recent information.

 

Update 1:53pm PST: Our technicians were unable to find anything wrong with the hardware and the machine has been online without issue for several hours now. We’ll be marking this as resolved, but please feel free to contact our Tech Support staff should you experience any further troubles. -Brandon

Previous updates available after the jump →


7:41 am

Shared server Andross on new hardware and restoring from backups ( RESOLVED )

Posted (June 9th, 2011 at 7:41 am PST) by brians

The shared server Andross was rebooted and came back up in read-only mode, which required a file-system integrity check, but was not able to pass the check successfully. We have completely replaced the hardware for this server and are now in the process of restoring from backups.

The server will remain up and running during this restoration process, as your user directories will be mounted directly from the backup server while we restore. We will be closely monitoring it for high load and taking action to keep sites accessible.

We sincerely apologize for any downtime you have already and may still experience. We appreciate your patience as we complete this restoration. If you would like email updates on this situation, please contact Support through your Web Panel.

This post will be regularly updated with the status of this process, so please check back here for the latest.

Update June 15th 11:20AM PST: Sorry for the delay in updating this post. It actually turns out that nothing needed to be done on the machine. Meaning, the restore / hardware replacement was not required. The old machine is fully functional and in tact. If you are still having issues, please submit a support request.

Previous updates available after the jump →


 
© 1996-2011, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
7:30 am

A Couple of Issues With New Domain Registration Offerings (RESOLVED)

Posted (June 9th, 2011 at 7:30 am PST) by mir

We’re in the process of correcting a couple of bugs that have cropped up with our new domain registration services:

-.uk domains require an ‘IPS tag’ which was not accounted for in our panel

-.es domain registration attempts are currently resulting in an error in our panel

-at least one .asia registration has triggered multiple confirmation emails when only one should have been sent

Our development team is working on clearing up these issues as quickly as possible. We’ll be sure to update this status post as the problems are addressed and resolved. -John

Update 2:55pm PST: We are still waiting on our upstream provider to provide feedback but for now .asia registrations are not available from the panel. As soon as we’ve been able to get that sorted out they will be visible again. -John

Previous updates available after the jump →


5:24 pm

CafeCommerce Scheduled Maintenance on June 22, 2011

Posted (June 7th, 2011 at 5:24 pm PST) by andrea

In order to keep your store as up to date and functional as possible, we must schedule periodic system updates.

On Wednesday, June 22, 2011, we will be updating the system between 10 ­and 11:00 p.m. PST. During this time, your CafeCommerce store, the CafeCommerce Manager and new CafeCommerce signups will not be accessible to either you or your customers. Please plan accordingly to reduce the impact this may have on you or your customers.

Thank you for your understanding. Don’t hesitate to contact support if you have any questions.

Update 10:24 PM PST: CafeCommerce stores came back up around 10:15. If you are having any issues, please contact tech support.


1:25 am

Shared server karpis going down for filesystem check (RESOLVED)

Posted (June 6th, 2011 at 1:25 am PST) by Ken

The shared server “karpis” is showing signs of possible filesystem errors. In order to ensure filesystem integrity, we are running a check on the home filesystem. Please be patient while we strive to improve your Dreamhost services.

UPDATE Mon July 06 0730 Pacific: Shared server karpis has finished its check and is now back in operation. If you experience any problems please contact Technical Support via the normal channels. We thank you for your patience while we continue to improve our services for you.


6:55 pm

VPS host machine homie-vserver41 down for unscheduled filesystem check. (RESOLVED)

Posted (June 5th, 2011 at 6:55 pm PST) by Jason C

Our admin team noticed some issues with homie-vserver41 and had to take it down for an unscheduled filesystem check. The servers that are affected by this outage are as follows:

ps10064 ps10066 ps10105 ps10300 ps10305 ps10307 ps10377 ps10609 ps12001 ps25269 ps25426 ps26140 ps26308 ps28525 ps29110 ps29127 ps29360 ps29878 ps30275 ps30301 ps34012 ps34840 ps34977 ps37655 ps42072 ps43572 ps44747 ps44836 ps49356 ps49584 ps49837 ps50164 ps50551 ps52295 ps52322 ps7618 ps8461 ps8462 ps8463 ps8464 ps8465 ps8466 ps8467 ps8468 ps8469 ps8470 ps8471 ps8472 ps8473 ps8474 ps8475 ps8476 ps8477 ps8478 ps8479 ps8480 ps8481 ps8482 ps8483 ps8484 ps8485 ps8486 ps8487 ps8488 ps8489 ps8490 ps8491 ps8492 ps8493 ps8494 ps8495 ps8496 ps8497 ps8498 ps8499 ps8500 ps8501 ps8502 ps8503 ps8504 ps8505 ps8506 ps8507 ps8508 ps8509 ps8510 ps8511 ps8512 ps8513 ps8514 ps8515 ps8516 ps8517 ps8518 ps8519 ps8520 ps8521 ps8522 ps8523 ps8524 ps8525 ps8526 ps8527 ps8528 ps8529 ps8530 ps8531 ps8532 ps8533 ps8534 ps8535 ps8536 ps8537 ps8538 ps8539 ps8540 ps8541 ps8542 ps8543 ps8544 ps8545 ps8546 ps8547 ps8548 ps8549 ps8550 ps8551 ps8552 ps8553 ps8554 ps8555 ps8556 ps8557 ps8558 ps8559 ps8560 ps8561 ps8562 ps8563 ps8564 ps8565 ps8566 ps8567 ps8568 ps8569 ps8570 ps8571 ps8572 ps8573 ps8574 ps8575 ps8576 ps8577 ps8578 ps8579 ps8581 ps8582 ps8583 ps8585 ps8587 ps8588 ps8589 ps8591 ps8592 ps8593 ps8594 ps8596 ps8597 ps8598 ps8600 ps8601 ps8602 ps8603 ps8604 ps8606 ps8607 ps8608 ps8609 ps8610 ps8611 ps8612 ps8613 ps8614 ps8615 ps8616 ps8617 ps8618 ps8619 ps8620 ps8621 ps8622 ps8623 ps8624 ps8625 ps8626 ps8627 ps8628 ps8629 ps8630 ps8631 ps8632 ps8633 ps8634 ps8635 ps8636 ps8637 ps8638 ps8639 ps8640 ps8641 ps8642 ps8643 ps8644 ps8645 ps8646 ps8647 ps8648 ps8649 ps8651 ps8652 ps8653 ps8967 ps8968 ps8971 ps8972 ps8975 ps8981 ps8987 ps8994 ps8996 ps9125 ps9127 ps9130 ps9131 ps9133 ps9142 ps9151 ps9318 ps9342 ps9540 ps9574 ps9579 ps9608 ps9831 ps9833 ps9842 ps9853 ps9876

Please know that we’re doing everything we can to get this matter resolved ASAP and we’ll keep updating this post as things progress.

Update 9:04pm PST: Our admin team has notified me that the filesystem check is complete and the VPS hosts should be coming back up. If you happen to see any errors, please contact our support team. – Jason

Previous updates available after the jump →


8:34 am

Panamacity server is temporarily down (RESOLVED)

Posted (June 5th, 2011 at 8:34 am PST) by danwy

The panamacity server is having issues and not coming up after a reboot. One of our admins is in the datacenter working on bringing it back up. We currently expect it to be back up and running within the hour. We will update this status post as soon as we have any more information or when the problem is resolved (whichever happens first). We apologize for any inconvenience this causes our customers and are working on bringing it back up ASAP.

Update: June 5th 9:55am PDT – The server is now back up and running. If your site is hosted on this server and you continue to have issues, please contact support and they’ll be happy to assist you further.


1:48 pm

Shared server portlouis down to be failed over to new hardware (RESOLVED)

Posted (June 4th, 2011 at 1:48 pm PST) by mir

The shared server portlouis is unresponsive and needs to be moved to new hardware. Our admin team is on the case and is in the process of handling this. We are expecting the process to take around an hour total from now. We’ll update this post as we have more information to report.

Update: June 4th 3:15pm PDT I apologize for the slow turnaround on the update but our admin team was working on the machine and unable to communicate but we have the details now. We thought we needed to fail the server over to new hardware but once we were able to get our hands on the server we found that the problem was the CPU so we just had to replace that and were able to get everything back up and running. The good news is that we don’t need to restore anything from backups (to new hardware) and everything should be resolved! -John


12:10 pm

DB Moves – Databases With Common Names On Specific MySQL Servers May Require Renaming

Posted (June 4th, 2011 at 12:10 pm PST) by Justin@DH

We have some older mysql servers from older, already retired clusters, that are aging and must be offloaded to new mysql servers to avoid any hardware failures from aging hardware, and for customers to receive newer, faster mysql servers. The affected servers are:

aiai anubis ape baby bandit barkley barnabus battlecat buddy buster butch cabbit catdog chimchim clarus clyde comet deputydawg diddykong dingo dodger drooper droopy duckworth felix flash furball gizmo gongon goofy gordo hotdog huan huckleberry indiana jingoro kingkong koko lady lily littleredmonkey madcat maverick mcgruff meemee missy mojo peabody pluto rafiki ragland rintintin roadrunner rosebud santaslittlehelper sassy shamu shiloh skimbleshanks streaky tib topogigo triumph wolpertinger

The new mysql servers are modern hardware, much more powerful and hold many more databases, which becomes a problem for moving customers who created databases before our more strict policy on database naming. Previously we allowed any name as long as it was not taken on that specific mysql server, but since a few years we have made it a requirement for names to be unique system-wide for new databases. Now these names may conflict on the new servers and require a rename to complete.

Customers who have a database with a very common name, such as:

– blog, wp, store, zencart, wiki, wordpress, gallery, pic, db, joomla, drupal, zc, etc

– also any customers with a dollar sign in their database name (something we have never allowed via the panel, plus it causes issues with moves, backups, restore, etc)

are encouraged to go to the control panel, in Goodies -> MySQL Databases, and click the rename database button next to the database, to avoid being affected later on. A good practice is to name the database after your domain, such as mydomain_com_blog. Afterwards any config files referring to the database, will need to be updated with the new name.

If we must rename a database, we will email your primary email address with the old and new name, so you are aware and can update your files.

Our apologize for requiring this change, and that we cannot email customers individually, as the rename only takes place if there is a conflict, and will not be required for all databases.

If you have any questions about this issue, please feel free to contact support.


10:49 pm

Downtime on a specific row of DEDICATED machines (RESOLVED)

Posted (June 3rd, 2011 at 10:49 pm PST) by brians

A row of DEDICATED machines at our Irvine datacenter will not reboot completely. Therefor you have likely noticed that your sites are offline. Again, only dedicated machines are setup on the row in question. Unfortunately we do not have an ETA at this time. I can assure you that our admin team is diligently working towards getting the machines back online. We will post more details as they are received. Sorry for any inconvenience this has caused. Your continued patience is greatly appreciated it. Please check back for future updates. -Brian S

ds1303 ds1385 ds1295 ds1302 ds1265 ds1267 ds1269 ds1272 ds1273 ds1274 ds1275 ds1278 ds1279 ds1280 ds1281 ds1282 ds1283 ds1284 ds1285 ds1286 ds1287 ds1288 d1289 ds1290 ds1291 ds1292

Update: June 3rd 10:50pm PDT Looks like this is effecting more users than we thought. The following list of servers are experiencing network issues, slow traffic, page loads, and ftp & shell access. We are working to get this resolved ASAP –Brian H

ds1265 ds1267 ds1268 ds1269 ds1271 ds1272 ds1273 ds1274 ds1275 ds1278 ds1279 ds1280 ds1281 ds1282 ds1283 ds1284 ds1285 ds1286 ds1287 ds1288 ds1289 ds1290 ds1291 ds1292 ds1293 ds1294 ds1295 ds1296 ds1297 ds1298 ds1299 ds1300 ds1301 ds1302 ds1303 ds1306 ds1309 ds1314 ds1316 ds1317 ds1319 ds1320 ds1321 ds1322 ds1323 ds1324 ds1325 ds1326 ds1327 ds1328 ds1329 ds1330 ds1331 ds1332 ds1333 ds1334 ds1335 ds1336 ds1337 ds1338 ds1339 ds1340 ds1341 ds1342 ds1343 ds1344 ds1345 ds1346 ds1347 ds1348 ds1349 ds1350 ds1351 ds1352 ds1353 ds1354 ds1355 ds1356 ds1357 ds1358 ds1359 ds1360 ds1361 ds1362 ds1363 ds1364 ds1365 ds1366 ds1367 ds1368 ds1369 ds1370 ds1371 ds1372 ds1374 ds1375 ds1376 ds1377 ds1378 ds1379 ds1380 ds1381 ds1382 ds1383 ds1384 ds1385 ds1386 ds1388 ds1389 ds1390 ds1392 ds1393 ds1394 ds1395 ds1396 ds1401 ds1402 ds1403 ds1404 ds1405 ds1407 ds1408 ds1410 ds1411 ds1412 ds1413 ds1414 ds1415 ds1416 ds1417 ds1418 ds1419 ds1420 ds1421 ds1422 ds1423 ds1424 ds1425 ds1426 ds1427 ds1428 ds1429 ds1430 ds1431 ds1432 ds1433 ds1434 ds1435 ds1436 ds1437 ds1438 ds1439 ds1440 ds1441 ds1442 ds1443 ds1444 ds1445 ds1446 ds1447 ds1448 ds1449 ds1450 ds1451 ds1452 ds1453 ds1454 ds1455 ds1456 ds1457 ds1458 ds1459 ds1460 ds1461 ds1462 ds1463 ds1464 ds1465 ds1466 ds1467 ds1468 ds1469 ds1470 ds1471 ds1472 ds1473 ds1474 ds1475 ds1476 ds1477 ds1478 ds1479 ds1480 ds1481 ds1482 ds1483 ds1484 ds1485 ds1486 ds1487

Update: June 3rd 11:28pm PDT We discovered a problem with one of the transceivers in our Distribution switches and have swapped it out. So far things have been holding steady for the last 20 minutes so we are marking this as resolved but will continue to keep an eye on the situation. — Sandon V.


7:33 pm

VPS Host, homie-vserver126 down (RESOLVED)

Posted (May 30th, 2011 at 7:33 pm PST) by JeffH

Homie-vserver126 failed to come back up after a powercycle.  We are dispatching an admin to fail over the machine as soon as possible.  ETA is estimated at 2 hours at this point.  We will update this bug as information becomes available.  We apologize for any inconvenience.

Affected VPSs:

ps19323 ps19327 ps19328 ps19330 ps19343 ps19344 ps19350 ps19381 ps19382 ps19384 ps19421 ps19426 ps19431 ps19524 ps19528 ps19554 ps19651 ps19655 ps19812 ps19939 ps19968 ps19984 ps20024 ps21037 ps21610 ps21642 ps21831 ps21850 ps23065 ps23651 ps26599 ps26720 ps30381 ps31267 ps32487 ps33572 ps33645 ps34564 ps34754 ps36820 ps43970 ps43975 ps43976 ps44410 ps45039 ps46286 ps46292 ps46736 ps47604 ps48404 ps48407 ps48512 ps48784 ps48786

Update 11:10pm PST: This server is now back up and running on new hardware! Please contact Support if you are still experiencing any issues!

Previous updates available after the jump →


4:55 pm

Shared server “fernandes” down (RESOVLED)

Posted (May 28th, 2011 at 4:55 pm PST) by Oscar

The shared server “fernandes” became unresponsive and has not come back up from a reboot. Our admin team is currently working on the issue and we will update this post as soon as we have more information.

Update 7:34pm PST: Server “fernandes” should now be back up and working. If you’re still seeing problems, please let support know! -Brian H


 
© 1996-2011, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
6:03 am

Shared server storm downtime due to reboot issues (RESOLVED)

Posted (May 27th, 2011 at 6:03 am PST) by patrickm

We apologize for the inconvenience this is causing, as the shared server storm is not coming up gracefully from a reboot. Our internal connections to the server are working, but the public interface is still down. Our Admins are working to resolve this issue and we will post here when that is done! Please contact Support with any questions!

Update 6:05am PST: The machine is back to normal and all services are back up. Please contact support if you are still experiencing issues with this server! – RobertP


3:37 pm

Discussion lists offline 5/26/11 @ 7:00PM PDT (RESOLVED)

Posted (May 25th, 2011 at 3:37 pm PST) by beth

We will be performing maintenance on our Mailman servers tomorrow, May 26th, 2011 @ 7:00PM PDT. During this time, discussion lists will be inaccessible. This process will take approximately one hour.

E-mail and files related to discussion lists (including configurations and archives) will not be lost. All discussion list e-mail will be queued until maintenance is complete, and delivered at that time.

This will only affect discussion lists. No other services will be affected.

Update May 26 07:43PM PDT: The scheduled maintenance is now complete and all discussion lists should be back online. -Beth


9:21 am

Vserver host machine Homie-vserver234 restoring from backups (RESOLVED)

Posted (May 25th, 2011 at 9:21 am PST) by Brandon

After a reboot of homie-vserver234, the file-system required an integrity check.  This check did not pass and the RAID array was not recoverable.  Due to this, our Admins are in the process of restoring this server from backups, using a brand new machine.

The following VPS guests are affected:

ps18424 ps18686 ps21403 ps22719 ps36138 ps36837 ps37718 ps37800 ps47390 ps49605 ps49609 ps49613 ps49630 ps49636 ps49638 ps49641 ps49654 ps49659 ps49692 ps49720 ps49744 ps49756 ps49767 ps49770 ps49791 ps49794 ps49828 ps49839 ps49847 ps49874 ps49922 ps49970 ps49974 ps50048 ps50141 ps50178 ps50179 ps50048 ps50141 ps50178 ps50179

Thank you for your patience with this issue.  We will regularly update this post with more information!

Update May 27 12:35pm PST: All VPS guest reconfigurations are done and the backup is completely done.  All VPS servers on the host machine are now back up. If you have any further issues with your VPS or sites being down, please contact  support staff and we will be glad to look into for you. -Oscar

Update May 27 11:16am PST: The restoration process is now complete and we are reconfiguring all guest VPS machines on the host machine. Once this is done, we will completely resolve this status post. -Oscar

Update May 27 9:12am PST: The restoration process is now at approximately 87% complete!  Everything is progressing along and we will continue to monitor the situation and update when this is nearing completion, or is complete!  -Brian S

Previous updates available after the jump →


9:58 am

MySQL server sawshark down for reboot (RESOLVED)

Posted (May 24th, 2011 at 9:58 am PST) by mir

Our admin team has been trying to determine the cause of high loads but was unable to troubleshoot due to performance so the machine is going down for a reboot and should be back up shortly at which time they will continue to observe and investigate. We’ll update this once the server is stable or we have more information to pass along. -John

Update May 24 10:10am PST: The server is now back up after performing a file-system check and services are active so this issue is resolved at this time. -John


1:46 am

Shared server johnnyblaze down for file-system check (RESOLVED)

Posted (May 24th, 2011 at 1:46 am PST) by patrickm

We apologize for the inconvenience this causes, but the shared server johnnyblaze did not come back correctly from a reboot and now requires a file-system check.  This process is to ensure the integrity of the data on the server and should take about 4-6 hours.  We will continually update this post with the status of this issue! -Peter

Update May 24 8:00am PST: All services are back up and running so this issue is resolved. -John

Previous updates available after the jump →


2:12 pm

Connection issues with MySQL servers (RESOLVED)

Posted (May 23rd, 2011 at 2:12 pm PST) by danwy

We are currently experiencing networking issues with some of our MySQL servers in the Alchemy datacenter. If you are getting a ‘Lost connection to MySQL server’ error on your site, please keep an eye on this post for updates. We don’t have many details on the cause of this error, but our admins are working on the situation now. We will update this post shortly once we have more information.

UPDATE May 23 3:40PM PST We’ve done extensive testing of the affected servers and have determined that things are back to normal now. We will continue to investigate the cause of this outage and make fixes wherever necessary to prevent this from happening again. If you continue to have issues with your sites please contact support, as it is not likely a result of this issue. We thank you for your patience. -DanW

Previous updates available after the jump →


5:31 pm

Homie-vserver129 failed to come back up after reboot. (RESOLVED)

Posted (May 20th, 2011 at 5:31 pm PST) by mir

What should have been a quick and simple power cycle of this machine is taking longer than expected to be concluded. Our admin team is in the process of getting it back up and running.

The following VPS guests will be affected by this maintenance:

ps19705 ps19713 ps19736 ps19961 ps20036 ps20097 ps20136 ps20409 ps20668 ps21645 ps22089 ps22111 ps22736 ps22737 ps23090 ps23103 ps23107 ps23326 ps23502 ps23571 ps23803 ps24019 ps24143 ps24185 ps24231 ps24904 ps25176 ps25396 ps25953 ps29070 ps29218 ps30253 ps30956 ps34839 ps36207 ps36766 ps36965 ps37253 ps37324 ps37326 ps37333 ps37377 ps37719 ps37913 ps38030 ps38050 ps38223 ps38745 ps38768 ps39314 ps39341 ps45476 ps46616 ps46920 ps46927 ps47153 ps47362 ps47729 ps48776 ps49060

We’ll update this post with a resolution or more information as soon as it becomes available. -John

UPDATE May 20 6:16PM PST All services have been restored and this issue is resolved, we thank you for your patience! -John

Previous updates available after the jump →


9:42 am

Homie-vserver169 is offline for RAID card replacement. (RESOLVED)

Posted (May 20th, 2011 at 9:42 am PST) by Brandon

Vserver host machine Homie-vserver169 will be going offline for  a short time in order to replace a faulty RAID card which has been contributing to recent stability issues on VPS guests hosted on the machine.

While we don’t foresee this taking very long we will keep this post updated with any new information as it becomes available.

The following VPS guests will be affected by this maintenance:

ps23351 ps23441 ps23716 ps23748 ps23789 ps23919 ps24075 ps24130 ps24304 ps24381 ps24655 ps24760 ps25029 ps25404 ps25466 ps25568 ps25569 ps25651 ps26006 ps26024 ps26058 ps26218 ps26335 ps26426 ps28499 ps28822 ps28824 ps28876 ps29041 ps29108 ps29160 ps29206 ps29313 ps30290 ps30419 ps30551 ps30655 ps31032 ps31128 ps33983 ps34010 ps34155 ps34434 ps34623 ps34841 ps35472 ps35844 ps36206 ps36220 ps36480 ps36573 ps36647 ps37671 ps37704 ps42219 ps42497 ps43595 ps43747 ps44219 ps47497 ps48020 ps48278 ps48280

Update May 20 11:48AM PST: The machine is now back online, and is looking stable. Should you experience any other issues, please don’t hesitate to drop a line to our Tech Support representatives. -Brandon

Previous updates available after the jump →


9:38 am

Shared server gretsch down for RAID card battery replacement (RESOLVED)

Posted (May 19th, 2011 at 9:38 am PST) by mir

The admin team expects this process to take around 15-20 minutes so we’ll update this post once we have more information. -John

Update May 19 10:00 am PST This has been completed successfully and the issue is resolved! -John


7:02 am

Shared server constantine down for file-system check (RESOLVED)

Posted (May 19th, 2011 at 7:02 am PST) by patrickm

The file-system for the shared server constantine became read-only, necessitating a file-system integrity check.  That is underway now and we hope to quickly recover the RAID on the server and bring it back online.

We will be regularly updating this post with new information on this issue.  Please don’t hesitate to contact Support with any questions you have!  We apologize for this inconvenience and we appreciate your patience!

Update May 19 9:15pm PST: The server has been plugging along for some time now and we feel the situation is stable enough to consider this issue resolved. If any further problems come up, you are always more than welcome to contact our support team. — Jason

Previous updates available after the jump →


 
© 1996-2011, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
6:39 am

Shared server roseau restoring from backups (RESOLVED)

Posted (May 19th, 2011 at 6:39 am PST) by patrickm

The shared hosting server roseau was rebooted when the file-system switched to read-only, which is a sign of an issue with the RAID array in the server.  The server did not pass the file-system integrity check, so were forced to replace the hardware for this server completely.  The backup server for the sites on this server is mounted as live data so that sites will be up and running during our data restoration process.

We will be regularly updating this post with as much information as possible.  Please contact Support with any questions you may have!  We sincerely apologize for the inconvenience and thank you for your patience!

UPDATE May 22 8:29PM PST The restore is now complete (and has been for a little while). The machine is now running locally again, using the last backup before the crash and any updates you may have uploaded since the restore started. If you’re still having issues with your sites on this server, please let our support team know with the details, so we can assist you.

Previous updates available after the jump →


4:38 pm

Roundcube webmail upgrade on 5/19/2011 (RESOLVED)

Posted (May 18th, 2011 at 4:38 pm PST) by andrea

We’re upgrading Roundcube tomorrow at 8pm PST. We’re anticipating an approximately 15-minute long downtime while we update the database and push out the code. We apologize for any inconvenience this may cause. We’ll update this post with the progress and when the upgrade is in place.

This will only affect you if you’re using Roundcube (not Squirrel Mail) for your webmail.

Update May 19, 08:51pm PST: Everything went without a hitch! The upgrade is done and has been working for a little while now. If anyone notices anything out of the ordinary, please contact our support staff and we’ll be glad to help you smooth things out.


1:34 pm

Private Server host ‘169’ is down for raid card replacement (RESOLVED)

Posted (May 18th, 2011 at 1:34 pm PST) by andrea

We apologize for the downtime on Private Server host ‘169’ ; the RAID card was having issues that caused the machine to crash. We are in the process of replacing this RAID card and hope to have the server back online shortly. Should anything go wrong, or hopefully right, we will update this post with details!

This host contains the following private server guests:

ps23351 ps23441 ps23716 ps23748 ps23789 ps23919 ps24075 ps24130 ps24304 ps24381 ps24655 ps24760 ps25029 ps25404 ps25466 ps25568 ps25569 ps25651 ps26006 ps26024 ps26058 ps26218 ps26335 ps26426 ps28499 ps28822 ps28824 ps28876 ps29041 ps29108 ps29160 ps29206 ps29313 ps30290 ps30419 ps30551 ps30655 ps31032 ps31128 ps33983 ps34010 ps34155 ps34434 ps34623 ps34841 ps35472 ps35844 ps36206 ps36220 ps36480 ps36573 ps36647 ps37671 ps37704 ps42219 ps42497 ps43595 ps43747 ps44219 ps47497 ps48020 ps48278 ps48280

Update 02:10pm PST: We’re still working on this and will provide further details as they become available. -John

Update 02:40pm PST: We are still in the process of replacing the RAID card and will update this post when we have further information. -Brandon

Update 02:55pm PST: All VPS guests on vserver host 169 are now back up and running! This issue is now marked resolved, but please contact our support team should there be any ongoing issues. -Brandon

Update 02:59pm PST: Unfortunately it seems that the machine has again crashed with the new RAID card. We will update this issue when more information becomes available. –Brandon

Update 3:20pm PST: We replaced the RAID card on the host machine again and now the server is booting back up. The first RAID card initially worked and then started to have issues. We are going to restart all guest machines now and will monitor the server for a few hours before we resolve the status post. –Oscar

Update 4:00pm PST: The host machine is down again and out admin team is working to get it back up. We will update this post as soon as we have more information. –Oscar

Update 4:30pm PST: The host machine is down again and out admin team is working to get it back up. We will update this post as soon as we have more information. –Oscar

Update 5:18pm PST: Unfortunately this machine is getting failed over. Our admin team is setting up a new machine right now and will start the fail-over process soon. We’ll update this post with more news once the fail-over starts and when the restore process has begun. –Oscar

Update 6:00pm PST: The new machine is up and the fail-over process has started. Once this is done, we will then start the restore process. We will keep you updated as we continue working on this issue. –Oscar

Update 6:30pm PST: Good news! This is actually going to be a chasis swap so we will not have to restore data. Essentially we are setting up a new machine and will be pulling the hard drives from the old server to the new server. This will help to reduce any further downtime. We’re still waiting on the new machine to be completely done with the fail-over process and will then be swapping disks. –Oscar

Update 7:10pm PST: The new server is now up and the disks have been swapped. We are reconfiguring all guest machines on the new VPS host machine. Some of your VPS machines are already back up along with your sites and the rest are coming back as the configs are finishing. We will update this again once the configs are all done. –Oscar

Update 7:50pm PST: The reconfiguration of all guest VPS machines is still going. If your VPS is still not up, it will be up shortly. –Oscar

Update 8:25pm PST: We’re still monitoring the progress of the guest reconfigurations as the last ones complete. We appreciate your patience with this situation and anticipate a final resolution soon. -John

Update 8:40pm PST: The reconfigurations of the guests have completed and all of the domains our admin team has spot-checked were up and running so this matter is being marked as resolved. -John


12:38 pm

Service queue is backed up (RESOLVED)

Posted (May 18th, 2011 at 12:38 pm PST) by danwy

Our service queue is temporarily backed up, causing automated updates to be delayed on our backend. This will affect things such as (but not limited to) user creation, domain additions, password updates, VPS reboot requests, etc. Basically any changes made via the panel that schedules an update to be made. This does not affect site or email performance. These changes will eventually be pushed out, they are just taking longer than normal. Our admin team is hard at work to resolve this situation and we will update this post as soon as we have any additional information.

Update 01:55pm PST: This issue should be resolved now and we’ve confirmed that updates are running smoothly again. There was a RAID consistency check running that caused things to get backed up. We’ve killed this process and that allowed the service queue to get caught back up again. If you continue to have issues with new services being activated, or existing services being updated, please contact support and they’ll be happy to assist you further. -Dan W

Update 03:30pm PST: While we believed the previous issues were resolved and the service queue was dropping, it seems that the issues returned and things are backing up again. Our admin team is already checking it out and hope to find the underlying cause. We apologize for the inconvenience this causes you and thank you for your patience as we try to resolve this issue. – Dan W

Update 05:31pm PST: The system is still processing the backlog of queued updates at a reasonable pace. Folks will still notice some delays, but hopefully they won’t be as bad. We’re expecting to make a bit more leeway in the next few hours as our US customers call it a night and stop adding updates to our queue. We’ll make sure to let you know what is going on tho. – Jason

Update 07:56pm PST: Things are still moving along and updates are being processed by our system. As of right now, the backlog is shrinking and we’re making forward momentum. – Jason

Update 08:25pm PST: We’re still making progress (we have about 10% less to process than we had at the last update and the improvement should accelerate as the night progresses. More updates to come. -John

Update 09:00pm PST: We’ve caught up a bit more than last time so things are progressing nicely – we’ll update this post in another hour. -John

Update 10:00pm PST: Another 20% less than last check so it’s speeding up. We’ll update again once additional significant progress has been made. -John

Update 11:25pm PST: Our admin team has confirmed the our service queue has returned to normal levels so we’re marking this as resolved. -John


8:23 am

Shared server Aquarius needs to be re-imaged (RESOLVED)

Posted (May 18th, 2011 at 8:23 am PST) by mir

The root file system on this server is corrupt. Fortunately the user files are unaffected so we just have to re-image the server to repair the root file system. We estimate the time to complete this fix to be around an hour and will update this post as we have more details. -John

Update 09:42am PST: The re-imaging is finished and we are configuring services now so everything should be fully operational within the next half hour. We’ll update this post once that is completed. -John

Update 09:58am PST: All of the services are configured and back up so the server is fully functional. This issue is resolved! -John

Update 09:59am PST: Scratch that – we’re fixing up some apaches right now – one more update to come. -John

Update 10:50am PST: Some of the apaches are up but we’re still configuring the rest so we’ll have to update again once we have more information. I apologize that this is taking much longer than anticipated. -John

Update 11:25am PST: About half of the Apache services on the machine are up and running without issue. We’re still working on bringing the rest online. -Brandon

Update 12:00pm PST: All of the Apache services are now running without issue, so this issue should now be completely resolved. As always, if you’re experiencing issues please feel free to contact our helpful Tech Support representatives! -Brandon


5:19 am

shared server ‘mogadishu’ down for RAID replacement (RESOLVED)

Posted (May 18th, 2011 at 5:19 am PST) by patrickm

We apologize for the downtime with the ‘mogadishu’ shared web server, as the RAID card was having issues that caused the machine to crash.  We are in the process of replacing this RAID card and hope to have the server back online shortly.  Should anything go wrong, or hopefully right, we will update this post with details!

Update 08:30am PST: mogadishu’s RAID card was replaced around 06:15am PST this morning and everything has been running just fine since so this matter is considered resolved.


8:42 am

Meissa Shared Webserver Downtime (RESOLVED)

Posted (May 16th, 2011 at 8:42 am PST) by Justin@DH

The meissa shared webserver is having slowness and downtime issues due to a domain that was receiving a large attack. The target of the attack has been dealt with and will no longer be causing issues on our network, but the traffic seems to be causing some issues despite being blocked at the core. Our network engineers will continue to work on it to stabilize the server. Our apologies for the problems this has caused. When we have further information we will update this post.

Update: Apologies for the lack of updates on this post.  This downtime was due to a Denial of Service Attack against one of the sites on this server.  This was resolved shortly after this post was made, but the post was never updated.  Thanks for your patience and please contact Support with any questions!


9:47 am

System maintenance on Wednesday at 2am PST (RESOLVED)

Posted (May 15th, 2011 at 9:47 am PST) by danwy

We are bringing the panel offline at 2am PST Wednesday 5/18/2011 for a brief maintenance to make changes to one of our central database servers. We expect the total downtime to be less than an hour but we are scheduling a downtime window of 2 hours. During this time you will not be able to access the panel and new service additions (such as adding users/domains/etc) will be delayed until the maintenance is done. We’ll update this post during the downtime and when the process is completed. We appreciate your understanding and patience during this scheduled maintenance.

Update 2:57am PST: This maintenance is now complete and all services are back online!  Please feel free to contact Support if you’re having any issues!


5:47 pm

Homie-vserver106 needs a new RAID card (RESOLVED)

Posted (May 14th, 2011 at 5:47 pm PST) by Brian@dreamhost

Homie-vserver106, a host machine for our Very Private Servers is going down for about 30 minutes. We’re doing this to replace a faulty RAID card that controls the server’s hard drives. We apologize for this, and hope to have it up and working ASAP! –Brian H

Affected machines are:

ps16217 ps16245 ps16248 ps16270 ps16271 ps16272 ps16289 ps16318 ps16320 ps16340 ps16341 ps16361 ps16377 ps16403 ps16441 ps16478 ps16528 ps16556 ps16697 ps16718 ps16721 ps16781 ps16786 ps16843 ps16850 ps17091 ps17127 ps17373 ps17383 ps17410 ps17438 ps17577 ps17925 ps18217 ps18320 ps18321 ps18336 ps18350 ps19779 ps19972 ps20212 ps20334 ps20699 ps21530 ps24791 ps38349 ps38352 ps38390 ps38513 ps38520 ps38523 ps38996 ps38998 ps39354 ps39924 ps40000 ps40024 ps45052 ps45056 ps45160 ps45516 ps45517 ps45808 ps46105 ps46287 ps46424 ps46779 ps47910

 

Update – May 14th -7:29pm (PST): The hardware update went well, but the kernel that is installed on the machine is having trouble supporting some functions it needs to start the virtual servers. We’re working on this and hope to have it updated soon!

Update – May 14th -10:33pm (PST): The host machine is now back up and all of your VPS’s should be back up now. If you are having any issues with your VPS or sites being down on this host machine, please contact our support staff and we will be glad to look into for you. –Oscar


2:03 am

WebFTP service down (RESOLVED)

Posted (May 13th, 2011 at 2:03 am PST) by patrickm

One of the central machines which handles our WerbFTP service (http://webftp.dreamhost.com) is down at the moment.  Our Admins are working on replacing the power supply for this machine.  When that is done and the service is back up, we will post that update here!  Thanks for your patience!

Update – 3:01am PST: This issue is now resolved!  The WebFTP service is working again.  Contact Support with any questions!


 
© 1996-2011, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
6:03 am

shared MySQL server ‘spurdog’ downtime (RESOLVED)

Posted (May 12th, 2011 at 6:03 am PST) by patrickm

We regret to report that the shared MySQL server ‘spurdog’ has crashed and we are having issues bringing it back up at this time.  Our Engineers are on-site and working to get this issue resolved as quickly as possible.  Assuming we can get this machine up without a file-system integrity check, it shouldn’t take more than an hour before it will be working again.

We will continue to update this post with the status of this downtime until it is resolved completely.  Please feel free to contact Support with any specific questions you may have!

Update – 6:11am (PST): The machine is up now. Please contact Support if you are still experiencing issues with this server.


10:20 am

LAX Networking Issues (RESOLVED)

Posted (May 11th, 2011 at 10:20 am PST) by Justin@DH

Our admins are continuing to see networking issues in a specific row of servers at our LAX data center. They are checking for any abuse of the network, such as attacks or DDOS. This network problem is unfortunately affecting a number of mysql servers in this row. Some of the servers affected include:

algernon alvin daisy dangermouse gus itchy jerry mickey mightymouse minnie nelly nicodemus paddington pukka ratbert simon sloop splinter stuartlittle theodore

This may have also been the cause of the outage earlier today for one rack of dedicated servers, posted here. – Justin

Update 11:03AM PDT: – We have restored networking to one rack, the servers sloop, alvin, theodore and simon have networking again and are working. We are still investigating further issues with the other racks and mysql servers listed above. – Justin

Update 11:55AM PDT: – The cause of the networking outage has been identified and dealt with, and will no longer be a cause of problems on our network. We had a lingering issue with the theodore mysql server, it has been rebooted and is working again. We believe all issues are resolved but will continue to monitor the situation. If you still see any issues related to this outage, please contact support. – Justin


7:39 am

Dedicated Networking Issues at LAX Data Center (RESOLVED)

Posted (May 11th, 2011 at 7:39 am PST) by Justin@DH

We are experiencing loss of network connectivity for a specific rack of dedicated servers at our data center. We have admins on the scene and looking into the issue. When we have more information we will update this post. The possibly affected dedicated servers, only a portion of those we have in service, are:

ds1245 ds1244 ds1247 ds1246 ds1249 ds1248 ds1251 ds1250 ds1252 ds1253 ds1184 ds1185 ds1186 ds1187 ds1188 ds1189 ds1190 ds1192 ds1193 ds1194 ds1195ds1200 ds1201 ds1202 ds1235 ds1236 ds1237 ds1238 ds1239 ds1240 ds1241 ds1242 ds1243

We apologize for the issues this morning and are working to get them resolved ASAP.

Update 7:56AM PDT: – Our admins have determined that the switch in the specific rack holding these dedicated servers has stopped functioning. One of our admins is heading to the datacenter with a replacement switch right now to swap it out.

Update 9:15AM PDT: – The network switch has been replaced, and service looks restored. We will be monitoring the new switch and servers for further problems. Our apologies for the issues this morning with networking.


8:23 pm

Mail users on “homiemail-sub3″ having issues with their accounts (RESOLVED)

Posted (May 10th, 2011 at 8:23 pm PST) by Jason C

One of the file servers that our “homiemail-sub3″ mail cluster mounts is currently having problems. Because of this, users on this cluster may not be able to send or receive mail via POP, IMAP, SMTP and Webmail. We currently have 3 MX (mail exchanger) servers that are exacerbating the issue and making this anything but an easy fix — but please know that we are working on this issue and will have updates here just as soon as our admin team lets us know what is going on.

Update – 9:26pm (PST): Everything should be back to normal as of now. This particular issue was caused by a faulty network cable on the filer that was being problematic. Swapping that out and letting the stacked up mail go out seems to have cleared things up for folks. If you continue to have issues with your mail, please don’t hesitate to contact support. – Jason


3:55 pm

Shared server “angel” down for file system check. (RESOLVED)

Posted (May 10th, 2011 at 3:55 pm PST) by Oscar

We are very sorry for the inconvenience this downtime is causing, as the ‘angel’ shared hosting server was rebooted and now requires a file-system integrity check to protect the data on the server.

Update – 5:26pm (PST): The file system check is going to take approximately 3 more hours to complete. We’ll update this post once we have more information. – Oscar

Update – 7:38pm (PST): The file system check is still chugging along. Currently, it’s at 56%. – Jason

Update – 9:09pm (PST): The file system check is getting closer to being done. Now we’re at 80%. – Jason

Update – May 11th – 1:35am (PST): The file-system check on this server is now complete and the server is back up and running!  Thanks so much for your patience during this process.  Please contact Support if you are experiencing any issues with your hosting.


5:50 am

Shared machine, “bishkek” is down for filesystem check (RESOLVED)

Posted (May 10th, 2011 at 5:50 am PST) by JeffH

We apologize for any inconvenience.  Upon reboot, bishkek’s filesystem went read-only and required a filesystem check.   The checks can take up to a few hours, but this depends on how many repairs are needed.  We will post back with updates as they become available.

Update – 11:00am (PST): Unfortunately  our initial filesystem check failed and the machine is not responding. Our admin team is at the data center working to get the machine back up and then we will run another filesystem check. This will take a few hours and we will update you as soon as we have more information.  -Oscar

Update – 11:40am (PST): The server is back up  and we are now running the filesystem check. We’ll update again once there is more information.  -Oscar

Update – 12:10pm (PST): Unfortunately the filesystem check failed. We have a standby machine already setup and are in the process of failing over the machine. We’ll update again once the restore begins.  -Oscar

Update – 1:50pm (PST): Unfortunately we’ve hit some issues with the new machine as well. Our admin team is trying to get the new machine to boot properly but it’s having some issues. We’ll update you again when the fail over gets started. -Oscar

Update – 3:10pm (PST): The  new machine is up now and the fail over process has started.  We’ll update again on the status of the restore. -Oscar

More updates available after the jump…


1:13 pm

‘kay’ server having its RAID card replaced (RESOLVED)

Posted (May 3rd, 2011 at 1:13 pm PST) by danwy

The server ‘kay’ has been having some issues with its RAID card, causing it to be intermittently unresponsive. We’re in the process of replacing its RAID card now and it should be back up and running shortly. We’ll update this status post once things are back to normal with the server. – danw

Update – 1:40pm (PST): The server has been back up and running for the last 15 minutes with the new RAID card and we haven’t seen any additional problems. We’ll continue to keep an eye on the server, but at this point we believe the issue is resolved. If you have continued problems on this server, please contact support and they’ll be happy to assist you further. -danw


3:03 pm

VPS host server homie-vserver231 down with fstab issues (RESOLVED)

Posted (May 2nd, 2011 at 3:03 pm PST) by Jason C

Our admin team is currently in the process of investigating issues with the VPS host server homie-vserver231. As it stands, one of its main drives was not coming up after an unscheduled reboot and the machine was having issues reading its fstab. This shouldn’t take long to fix, but we wanted to let the customers affected by the issue know ASAP.

If your server is in the list below, you’re one of the unlucky few:

ps10181 ps17300 ps36800 ps43559 ps47507 ps9584 ps9893 ps9934

We should have some good news for y’all soon — but if there are complications, we’ll be sure to update this post accordingly.

Update – May 2nd, 3:36pm (PST): And we’re back! The issues with the host server have been sorted out and everything is running smoothly. If they stay stable, we’ll mark this post as resolved around 4:30pm (PST). – Jason


5:17 pm

Panel maintenance on Wednesday at 2am PST

Posted (May 1st, 2011 at 5:17 pm PST) by andrea

We are bringing the panel offline at 2am PST Wednesday 5/4/2011 for a brief maintenance to fix a problem with one of our central databases. We’ll post more details as soon as we can.

More details: The downtime is estimated to last less than 30 minutes. We’ll update this post during the downtime and when the process is completed. We appreciate your understanding and patience during this scheduled maintenance.

Update 2:10am (PST): This maintenance was completed swiftly with minimal downtime!  Please contact Support if you have any issues, or you still cannot access your Customer Web Panel!


9:27 pm

Nitti raid issues – restoring from backup (RESOLVED)

Posted (April 30th, 2011 at 9:27 pm PST) by Justin@DH

The nitti server had a severe raid issue that required that we move the machine to new hardware. We have the backup server mounted on it and it is restoring from backups. The server should be working and up for customers during the restore, since it has the backup live for websites to be live. When we have further info on the progress of the restore we will update this post.

Update – 11:57pm (PDT): The server has been about 70% up as of about an hour ago but there were some additional complications that prevented the machine from service being completely restored. As of about 10 minutes ago all services are restored on this machine.

Update – May 1st, 7:30am (PST): The nitti server appears to be doing well and is stable during the restore. The restore is currently 6.5% done.

Update – May 2nd, 8:34am (PST): The nitti server restore is going well. It is currently 35% restored.

Update – May 3rd, 9:11pm (PST): We’re still restoring data on the server. The current progress is 70%. – Jason

Update – May 4th, 12:59pm (PST): Data is still being restored on the server without any issues. The progress is currently at 81%. – Jason

Update – May 4th, 6:17pm (PST): The restore presses on. Currently, progress is at 91%. – Jason

Update – May 5th, 12:30pm (PST): The restore is now complete. Thank you for your patience, and please contact support if you need any help with your sites on Nitti. – Andrea


 
© 1996-2011, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
2:37 pm

Major System-Wide Outage (RESOLVED)

Posted (April 30th, 2011 at 2:37 pm PST) by Josh

We are investigating the cause and will update this post as soon as we know more.

Update 2:54pm PST: One of our network providers is having issues right now. Our network admins are working on moving all traffic to our two other service providers. We will keep updating this post as soon as we have more information. -Oscar

Update 3:10pm PST: We are having issues with one of our transit links to our LAX datacenter, our network administrators are investigating the issue and we hope to have it resolved as soon as humanly possible. This is causing problems with both external and internal networking – we’ll update this post as soon as we have more information or the issue is resolved. -John

Update 3:30pm PST: Our network engineers are on site at the source of the issue and working on resolving it. More information to follow as it becomes available. -John

Update 3:50pm PST: We are still working feverishly to resolve this issue – some portions of our network are responding better now but there’s more work to be done. -John

Update 4:10pm PST: We are seeing improvement but are still working on this issue. -John

Update 4:40pm PST: Our network engineers rebooted the core router at our LAX facilities but our providers are still flapping. It appears that the LAX facility may be stabilized but we’re experiencing high load on our core routers as a result of all the BGP flapping. We are still investigating the issue but current traffic patterns possibly indicate an attack on our network. -John

Update 5:00pm PST: We are still working to defeat this attack on our network. Nullifying certain source IPs has seemed to help matters, but that attack vectors and profiles are still being introduced and identified. -John

Update 5:25pm PST: Some attacks have been blocked that were inbound and we have identified some that were outbound and blocked those as well. We’re still working on the matter and the network responsiveness is improving but the attacks are still causing BGP flapping. -John

Update 5:50pm PST: We’ve pinned down the sources and nature of the attacks and are in the process of scanning for more attack software as well as filtering bad traffic at the core routers. The issue is not completely resolved yet but we’re making good progress now. More updates to follow. -John

Update 6:15pm PST: In a nutshell we suffered an extremely sophisticated attack. It took a while to get things under control enough to see what was going on and then start not only blocking attack vectors but track down and disable software being used to launch attacks from our network as well. Things are almost under control currently and once they are we’ll get a full and detailed report from our network engineers for you. -John

Update 6:35pm PST: Good news – all inbound attacks appear to now be screened and the CPU levels on our networking equipment is currently stable. We’ve strengthened our defenses at all levels of our network as well – borders are hardened, cores are protected, transit interfaces are stable. That said, we’re remaining vigilant as these attacks are much more advanced than what we’ve seen previously. -John

Update 6:50pm PST: All indications are that everything is under control so we are marking this matter as resolved. We will still provide details on what happened as soon as we’re able to do so. -John

Update 7:12pm PST: It’s back.. we’re on it again.. more updates soon. -Josh

Update 7:31pm PST: It seems that we’re seeing issues with our transit links to our Alchemy datacenter. Our network engineers are working on the issue and we will update this as soon as we have more information. – Oscar

Update: 8:02pm PST: We are seeing some more interruptions in networking, our admins and network engineers are still looking into the issue. – Justin

Update: 8:25pm PST: Our abuse team is getting involved as well at this point. We have resolved issues on most of our routers, and have found over 500 compromised sites so far, that we are working on fixing. Overall things should improve over time. -Justin

Update: 8:53pm PST: Things once again seem to be resolved.. we’re going to be cautiously optimistic and say nobody should be seeing any network issues at this time. We’re still going through squashing bad guys here and there, but the system is now sound. We’ll update this post if anything changes. -Josh

Update: 9:14pm PST: I spoke too soon again.. we’re still on it. -Josh

Update 9:41pm PST: Okay, it seems like we were being attacked because of a site we host. We’re investigating, things should be mostly back to normal for nearly everybody but the attack continues.. -Josh

Update (May 1) 11:50pm PST: This issue is officially resolved.  Please contact Support with any questions!


11:48 am

shared server ‘managua’ down for file-system check (RESOLVED)

Posted (April 30th, 2011 at 11:48 am PST) by Oscar

We are very sorry for the inconvenience this downtime is causing, as the ‘managua’ shared hosting server was rebooted and now requires a file-system integrity check to protect the data on the server.

Update – Apr 30, 1:05pm PST: Great news! The raid card was replaced and the server did not boot in read-only mode. No fsck is required and the server is back up along with your sites. If you are having any further issues with your sites, please contact our support staff and we will be glad to look into for you.

 


11:12 am

Shared web server aquila is having raid issues (RESOLVED)

Posted (April 29th, 2011 at 11:12 am PST) by Oscar

We are sorry to report that the shared server ‘aquila’ crashed and upon reboot the raid setup no longer responded. One of our admins is on his way to the datacenter to setup another machine and start the restore process. The ETA for this to be done is a few hours. We will update this post as soon as we have more information.

Update: 12:40pm PST: Great news! We were able to swap out the raid card on the server and it rebooted just fine! All of your sites are now back up and resolving. Should you have any further issues with your sites, please contact our support staff and they will be glad to look into for you.

 


7:59 pm

Shared server ‘eskimos’ back down for file system check. (RESOLVED)

Posted (April 28th, 2011 at 7:59 pm PST) by Jason C

We’re rather sorry to report that eskimos has had some issues and had to go back down for a file system check. This means that sites on the server will be inaccessible for the time being.

Since this is now the 3rd file system check in as many days, the server has become a prime candidate for being failed over to new hardware. If this happens, we will update this post with all of the details. We’ll also make sure to let you know how far along the file system check is as well.

Please know that we’re terribly sorry for any issues this might cause and we’ll do everything we can to resolve this matter quickly.

Update – Apr 28th, 9:31pm (PST): Our admin team is now in the process of failing the server over to new hardware. This will improve stability moving forward, but the outage on the server will be extended by the migration. We’ll give you more updates as this move progresses.

Update – Apr 28th, 10:05pm (PST): The new hardware is in place and the restore from backups is beginning now.  As usual, we will be mounting the backup server in such a way that we can bring the server back online and start serving sites again.  The actual restoration will take some time, but services will be restored during that process.  We will update this post again shortly!  We thank you for your patience during this downtime.

Update – Apr 29th, 12:23am (PST): Apologizes for the lack of update these past few hours.  Our estimate on when the server would be back online and serving sites was off, as we encountered an issue with the first array we created for the new server.  We are now on track and expect the server to be up and running within the next 30 minutes.  Note that the restoration will take longer, but sites should be coming up shortly!

Update – Apr 29th, 1:40am (PST): We are happy to report that the server is now up and running, as we have just finished bringing all the Apache services back online.  The restoration of data is also running, so there may be short periods of high load on the server.  Our team will be monitoring the server during this restoration to ensure that we provide the most uptime possible while the process completes.

We will continue to update this post with details on the restoration of data to this server.  Please feel free to also contact Support if you have any questions or issues!

More updates available after the jump…


12:45 am

shared server ‘professorx’ downtime for hardware replacement (RESOLVED)

Posted (April 28th, 2011 at 12:45 am PST) by patrickm

We regret to report that after a failed reboot of the shared web server ‘professorx’, we are now completely replacing the hardware on the server.  This process should take about an hour to complete and we will update this post when that is done!

Update 01:15 PST: This work has been completed and the server is now back up and running on new hardware!  Please contact Support with any questions, or issues you might have!


11:10 am

Shared server ‘Eskimos’ down for file system check. (RESOLVED)

Posted (April 27th, 2011 at 11:10 am PST) by Brandon

One of our shared servers (Eskimos) was displaying some possible file system issues and we’ve taken it down for a file system check. Typically this takes several hours to complete and any websites running on this machine will not be accessible until it has finished. We will update this post with further information as it becomes available, and sincerely apologize for any inconvenience this has caused and appreciate your continued patience.

Update – Apr 27th, 11:52am (PST): The first pass is still running as expected and there have not been any issues with the file system check.

Update – Apr 27th, 1:29pm (PST): The file system check is still humming along without issue.

Update – Apr 27th, 2:54pm (PST): Eskimos is still having it’s file system checked and things are running smoothly

Update – Apr 27th, 5:04pm (PST): Eskimos is still having it’s file system checked and it’s at 50%. We’ll update this again shortly.

Update – Apr 27th, 6:40pm (PST): The file system check is at 99%.  It’s almost done, and we will give the all clear.

Update – Apr 27th, 6:50pm (PST): Finally! The file system check is complete! Apache services have started and our admin team has given us the “all clear”. If you should happen to notice any issues with your server, please contact our support team and we’ll be more than happy to help.


5:29 pm

Shared web server skopje is having its raid card swapped (RESOLVED)

Posted (April 26th, 2011 at 5:29 pm PST) by andrea

This is an emergency hardware fix, sorry about the lack of notice. The expected downtime is around 15 minutes. This is happening now, and we’ll update this post as soon as it’s done or if anything comes up.

UPDATE 18:10pm PST: As of about half an hour ago, the server came back online and appears to be serving sites correctly. If you’re on skopje and having issues with your sites, please contact support with the details. We apologize for any inconvenience this downtime has caused.


6:09 pm

Discussion list URLs timing out (RESOLVED)

Posted (April 23rd, 2011 at 6:09 pm PST) by mir

Our admin team is fixing an issue with DNS which is causing some discussion list addresses to time out when trying to access them via a browser.

Instant update: the admin team has confirmed that the problem is fixed but as it was due to DNS the fix has not gone live. It should be working in the next hour or two when that completes.

Update – Apr 24th, 10:20am (PST): We are still experiencing some issues with discussion list DNS information. Our admin team is still working on sorting it all out and we hope to have it back to normal soon. We will update this post when we have any additional information.

Update – Apr 24th, 12:16pm (PST): We have word from our admins that the issue should be resolved. If you still have issues, please contact support.


6:16 am

DNS updates delayed, several hours behind. (RESOLVED)

Posted (April 23rd, 2011 at 6:16 am PST) by JeffH

Last night during a network maintenance window, which was anticipated to be transparent from the customer end,  our slave DNS machines lost connection to our master DNS MySQL server.  This means that our DNS is currently running off of old data (~11:00pm last night), and newly created DNS records and changes to DNS will not take effect until the slave machines catch up.    This could possibly take as long as 10 hours for everything to settle.  The only services which should be affected would be DNS changes since last night at 11:00pm.  If there are any other issues, please contact support and we will be happy to assist you!  We apologize for any inconvenience this may cause.

Update 2011-04-23 15:30 PST: DNS updates should be all caught up now, as the main DNS server’s mysql instance has resync’d it’s data. There may still be some lag, however, as our second and third name servers are still catching up. We’ll update with more info, as we get it!

Update 2011-04-26 17:30 PST: This has been resolved as of Sunday. Please contact support if you need assistance, as it’s most likely unrelated.


3:22 pm

Servers scheduled for BIOS updates on April 20, 2011 @ 5:00pm (PST) (RESOLVED)

Posted (April 20th, 2011 at 3:22 pm PST) by Jason C

Some time after 5:00pm (PST) today a number of machines in one of our racks will be going down for a brief period of scheduled maintenance. This is so a member of our admin team can update some settings in the BIOS to help improve remote management. We know this may be sort of a hassle for some folks, but improving remote management is something that will benefit every customer.

Downtime should run no longer than 15 minutes per machine and the servers will be worked on one at a time to minimize issues. We will do our best to button this up by 10:00pm (PST) but if we do run over, we will update this entry accordingly.

The batch of machines scheduled for maintenance are as follows:

Shared Servers: europa callisto himalia amalthea neith chiron themis theia mebsuta megrez meissa mekbuda lesath maasym mahasim maia matar marfik marfark markab

Virtual Private Servers: ps48565 ps48603 ps48615 ps48619 ps48629 ps48641 ps48645 ps48646 ps48648 ps48656 ps48660 ps48663 ps48672 ps48674 ps48677 ps48683 ps48686 ps48688 ps48694 ps48695 ps48699 ps48702 ps48703 ps48705 ps48712 ps48713 ps48714 ps48719 ps48721 ps48727 ps48728 ps48730 ps48733 ps48736 ps48738 ps48566 ps48599 ps48604 ps48618 ps48621 ps48623 ps48626 ps48628 ps48630 ps48631 ps48636 ps48638 ps48642 ps48652 ps48657 ps48661 ps48662 ps48664 ps48666 ps48668 ps48673 ps48678 ps48680 ps48685 ps48697 ps48698 ps48704 ps48711 ps48717 ps48720 ps48724 ps48734 ps48737

P.S. Sorry about the minimal warning on this one, but the upgrade has been deemed absolutely necessary by our admin team.

Update 23:45 PST: This all went smoothly!  We actually finished this around 17:00 PST, so if you are having any issues currently they are likely unrelated.  Please contact Support in that case and we will check it out!  Thanks again for your patience with these updates!


 
© 1996-2011, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
11:59 pm

shared server ‘gambino’ downtime and fschk (RESOLVED)

Posted (April 19th, 2011 at 11:59 pm PST) by patrickm

We are sorry to report that the shared server ‘gambino’ crashed and upon reboot requires a file-system integrity check.  This process can take some time, but we expect around 2 hours of downtime.  We will be updating this post with any details we have and if this ETA should change as we progress along!  Thanks for your patience and please feel free to contact Support with any questions you have!

Update 03:45 PST: The downtime from this was actually much less than expected. The file-system check came back quickly with a clean check on each drive.  However, after several reboots of the server, the check was still coming up, so we replaced the RAID card in the server just to be sure.  We monitored the server for some time to make sure there were no more issues.

As always, if you notice any issues with your hosting, please contact Support!


1:14 pm

VPS host ‘homie-vserver02′ down for file-system check (RESOLVED)

Posted (April 15th, 2011 at 1:14 pm PST) by Brandon

One of our host machines was displaying some possible file system issues and we’ve taken it down for a file system integrity check. Typically this takes up to several hours to complete and any VPS guests running on this host machine will not be accessible until it has finished.

This list includes all affected VPS guests:

bcboo cardamom clzvps creativesource cybercomuk cynsspot eglobal gauss golfzone goog1 indev liupoyuan lunari nbacharters nevin pastelserver pbbnz peregrine ps10220 ps11035 ps11106 ps12206 ps12921 ps12963 ps13138 ps13184 ps15476 ps15582 ps15834 ps22524 ps28703 ps29920 ps35750 ps35755 ps35789 ps36323 ps37168 ps37170 ps37184 ps37907 ps38455 ps38553 ps39054 ps39991 ps40052 ps4164 ps4171 ps42276 ps44228 ps44881 ps4530 ps46308 ps46933 qubitron rockingromans stevemccurry tastespotting tyax urbansonar voicer01

 

 

We sincerely apologize for any inconvenience this has caused and appreciate your continued patience.  We will update this post as soon as we have further information.

 

Update 1:42pm PST: The first pass is at 54%, however there will need to be a second pass run when this has finished.  We’ll update this post shortly.

Update 2:35pm PST: The first pass has completed and the second pass is at 80% completion.

Update 2:42pm PST: The second pass has completed. The VPS host machine is back online and all guests are up and running. Should you still be experiencing any issues, please contact our Tech Support staff and they’ll be glad to assist.


3:45 pm

Webmail problems (RESOLVED)

Posted (April 14th, 2011 at 3:45 pm PST) by danwy

Some of our webmail machines are having issues, we believe, due to NFS. Our admin team is currently working on it and hope to have it back to normal shortly. We will update this status post when we have more information. We’re sorry for any inconvenience this causes you.

Update – 3:50pm (PST): It looks like this issue is already resolved. One of our file servers caused a few of the webmail machines to become unresponsive. We’ve rebooted them and made sure there were no further mounting issues. We’ve tested several webmail accounts and they are all working now. If you continue to have issues though, please contact support to have them investigate it further.


5:10 am

shared server ‘vaduz’ downtime for RAID maintenance (RESOLVED)

Posted (April 14th, 2011 at 5:10 am PST) by patrickm

We are sorry to report that the shared server ‘vaduz’ is being taken down to address issues with the RAID controller and to complete a file-system integrity check.  We do not have an ETA on when this will be completed, but we will update this post as soon as we have a better idea.  If you have any questions about this, please contact Support!

Update: Sorry about the lack of update on the resolution of this issue. The problem was resolved around 10am pacific time yesterday, so roughly 5 hours after this post was initially created.


3:17 pm

Shared web server nuuk is having hardware issues (RESOLVED)

Posted (April 13th, 2011 at 3:17 pm PST) by andrea

We’re currently looking into resolving the hardware problem we found while doing a routine check on shared web server ‘nuuk’. We apologize for the downtime, we’ll update this post as soon as there is progress.

UPDATE 15:57 PST: Nuuk should be back up soon, thanks for your patience. We’ll update this post again when we are able to test and make sure the machine is stable.

UPDATE 16:44 PST: Nuuk is now back up and stable. We tested all the reported sites, and they are loading fine. If you’re still having issues with your sites on nuuk it’s likely unrelated, so please contact support through the panel with the specifics, and we’ll be happy to help.

UPDATE 18:00 PST Unfortunately, nuuk is again experiencing some issues. We’re looking into it, and will update this post with any progress. Sorry about the inconvenience this is causing.

UPDATE 20:00 PST  The raid card has been replaced on the server and it’s now back up. All of your sites should now be working. If you are still having trouble accessing your site, please contact our support staff and we will be glad to look into for you. Our sincere apologies for the downtime.

 


6:19 am

Managua webserver having raid issues (RESOLVED)

Posted (April 13th, 2011 at 6:19 am PST) by Justin@DH

The managua webserver, which previously had issues where its raid needed a file system check, went unresponsive and is having issues to boot up and detect its raid array. One of our server admins is on his way to replace the raid card, which is the first step in debugging and fixing such hardware issues. When he have further news on the progress of the repair, we will update this post. Our apologies for the repeat issue that this specific server is having.

Update: 2011-04-13 7:06:00 – The raid card is being replaced now. Please standby for an update after the card has been replaced.

Update: 2011-04-13 7:36:00 – The raid card is replaced, the server running normally at this time and all services restored. If you are still seeing any issues please contact support.


12:24 am

shared server ‘ibanez’ down for hardware replacement (RESOLVED)

Posted (April 13th, 2011 at 12:24 am PST) by patrickm

We sincerely apologize for this, but the shared server ‘ibanez’ is down at this time due to hardware issues.  Our Admins are replacing the server hardware at this time and hope to have it back up and running shortly.  We will update this post with details, but please feel free to contact Support with any questions you may have!

Update 12:42am PST: Great news!  Turns out we did not need to replace the hardware on this server!  A quick flash of the BIOS on the motherboard has cleared up the boot issue we were seeing.  The server is coming up now and all services are restored!  Please contact Support with any questions or issues!


5:21 am

shared server ‘managua’ down for file-system check (RESOLVED)

Posted (April 11th, 2011 at 5:21 am PST) by patrickm

We are very sorry for the inconvenience this downtime is causing, as the ‘managua’ shared hosting server was rebooted and now requires a file-system integrity check to protect the data on the server.

We will update this post when this process has been completed!

Update: Apr 11 8:39AM – The file system check has completed quicker than expected, there is less data than normal on this server, and service has been restored. If you see any services still not working on this server please contact support.


2:03 pm

shared server ‘moran’ needs another file-system check (RESOLVED)

Posted (April 10th, 2011 at 2:03 pm PST) by danwy

The shared server ‘moran’ is down for another file-system integrity check. This is required when rebooting the server to ensure that no data is lost. We don’t have an ETA at this time on when the issue will be resolved, but we will post updates about it here.

Update @ 7:49pm PST: Due to all of the issues with this server, our admin team is setting up a second machine and populating the hard drives with a recent backup. Once the original server is back from its file system check, our admins will begin syncing the new server (pre-populated with some recent content) with the original server (with the most up to date content).

Hopefully, this will make the transition to the new hardware as pain free as possible. More updates coming soon!

Update 04.11.2011 @ 1:28am PST: This server is now running on new hardware!  Data is being mounted from backups while we completely restore the data on the new server from the most recent backups.  This means services will be up, but may be slow at times due to the load on the server.  Please feel free to contact Support if you’re having any issues!  We will keep this post updated as we complete this restoration!

Update 04.11.2011 @ 10:13am PST: We had to turn off the apache web services on the machine for the period of the data restore, as they were destabilizing the machine and putting the restore at risk. We’re running the restores as fast as we can, so we can soon turn back the web service, but for now, we do not yet have a time estimate. We’ll update this post again as soon as we have more information. Very sorry about the inconvenience this is causing.

Update 04.11.2011 @ 10:56am PST: Based on the current rate of data copy, we’re estimating the restore done in about 30 hrs. Unfortunately, we have to keep the apache web services turned off to keep the machine stable in the meantime. Again, apologies for the inconvenience.

Update 04.11.2011 @ 14:52 PST: Currently, we have restored 200GB out of 2.8TB and are continuing to monitor the progress. We also turned back on some of the apache web services to stress-test the machine, and so far so good. We’ll update this post again soon with more information. Please contact support through the panel if you have any specific questions regarding your domains.

Update 04.11.2011 @ 16:13 PST: We’re at 218 GB restored right now, and all apache web services are up and running. The machine continues to remain stable while also keeping up with restores. Your sites should be up and running now, so let our support know through the panel if you need help. Again, you may update your data during this restore process, as it won’t be overwritten.

Update 04.11.2011 @ 18:03 PST: The restore is chugging along, and the server is remaining stable with all apache web services turned on and running. If your site is still not working, or not working as expected, please note that if you’ve updated the site since our last backup (which may be the missing puzzle for it to work), you will have to create/upload those changes again. Also, we don’t back up any /cache directories, so if you see errors on your site related to that, it’s something to keep in mind. Please contact support through the panel if you need further assistance.

Update 04.11.2011 @ 19:44 PST: Some Apache services are being a bit problematic. Our admin team is working on keeping them up thru the increased server load caused by the restore. Hopefully they should stabilize soon.

Update 04.11.2011 @ 21:18 PST: Our admin team continues to try to keep the Apache services on moran stable. As of this moment, we’ve got 245 GB of 2.8 TB of data restored to the server. Things are moving a bit slower than we would like, but we want to make sure the server is accessible to our customers on it.

Update 04.12.2011 @ 03:00am PST: This process is still underway and we are at around 333GB restored so far.  We are constantly monitoring the load on the server to keep it low, such that services remain active.

Update 04.12.2011 @ 11:30am PST: We’re at 433GB restored, and the admins are looking into speeding up the restore rate. In the meantime, the server continues to remain stable. Please let support know through the panel if you need any assistance with your sites on Moran.

Update 04.12.2011 @ 14:33pm PST: The amount transferred from our backups is now at 618GB and things are staying stable. As always, support is available for any questions you may have about this matter — so don’t hesitate to contact us!

Update 04.12.2011 @ 21:07pm PST: The restore has reached 722GB at the time of this update. That’s not bad, but the load is a bit higher than we’d like. Our admins have been tapped to try to get it back down to something reasonable. As always, we have tech support standing by if you have any questions.

Update 04.13.2011 @ 0:700 PST: The restore is at 823G currently so we’re about half way through now, more updates to follow.

Update 04.13.2011 @ 10:44PST: A bit of correction on the previous post: the current restore rate is at 25% done (right now at 929GB). In any case, the server is running fine and all sites should be up. Keep checking back for updates, but if you need any specific help, contact support through the panel.

Update 04.14.2011 @ 05:30 PST: Quick update! We are now at 1.5T out of 2.8T restored!  This means we are actually more than 50% complete now!  The load is remaining really low on the server and services are up at this time.  Please contact support if you are experiencing any issues!

UPDATE 04-14-2011 13:45 PST: Just a quick update. At the moment 7.8M out of 18M files have been copied ( M = million ). Just to keep it consistent with previous posts ( 1.6T of 2.8T ). However in this case of this particular restore, the number of files copied is more important than the actual size. So we are just under half way point. Sorry again for any inconvenience this has caused. Your continued patience is greatly appreciated. Please check back for updates.

Update 04.15.2011 05:00 PST: Morning update!  The process is continuing and we now up to 12M out of 18M (M = Million) files restored!  That’s 1.4TB out of 2.8TB.

UPDATE 04.15.2011 1:55PM PST Afternoon Update! Restoring is still chugging away. We are up to 15M out of 18M. Which also equates to 2.5T out of 2.8T. The finish line is approaching! Thanks again for bearing with us. Please check back for updates.

UPDATE 04.16.2011 1:55PM PST Afternoon Update! The restore is finally done and all user/ website data has been restored from our backups. If you are still having issues with your sites being down, please contact our support staff and we will be glad to look into for you.


6:39 am

IPv6 Connectivity Restored

Posted (April 10th, 2011 at 6:39 am PST) by Justin@DH

We had a couple reports of issues connecting with ipv6 to our network, and after investigating we found the cause and it has been corrected. IPv6 is slowly coming into more use, but at this time is used less often than IPv4 so it is unlikely to have caused any noticeable difference for the vast majority of our customers, but it is the future of the internet and it working properly is important. Our apologies for the issue.


 
© 1996-2011, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
1:31 pm

SSL Certificate orders not being processed. (RESOLVED)

Posted (April 8th, 2011 at 1:31 pm PST) by David

First of all we sincerely apologize for any inconvenience this may cause.

Due to some changes implemented by our Certificate Authority partner some new and renewal SSL certificate orders are not going through. Part of the new procedure is that ALL SSL certificate orders now require Domain Control Validation (DCV) before the orders can be approved. That this means is that an approval e-mail is sent to a specific e-mail address for that domain that only an administrator should have access to. The e-mail contains a link and instructions to approve the order so it can proceed to be processed by our Certificate Authority.

If you have submitted a SSL certificate order prior to 4/5/2011 (either manually or an auto-renewal) that has not already been completed (still shows status “ON ORDER” in our control panel under (DOMAINS > SECURE HOSTING)) then we may need to cancel that order, process a refund and have you initiate a new order in our control panel.

However, if you have initiated a SSL certificate order and were prompted for a DCV address during the ordering process then these orders should be processed without any problems. Once the approval instructions have been completed in the DCV e-mail that you received the order should be processed normally and will be installed automatically in our system as soon as the order is completed.

Our developers are working on an update to our system that must be completed before these new orders can be placed and we will update this post as soon as it’s ready to go.

Again we sincerely apologize for any inconvenience this may cause. We know how important secure hosting is to our customers and are working to get this corrected as quickly as possible. Please check this page for updates.

UPDATE 2011-04-08 3:51PM PST

Everything has been resolved. Our developers have pushed out the updated code. New orders and renewals will work properly now, and the auto-renew will now be sending an e-mail letting you know to manually renew the certificate in the panel. We sincerely apologize for any inconvenience this has caused and appreciate your continued patience. If you require further assistance, please submit a support request through our Panel.


8:14 am

Gambino webserver downtime due to hardware issue

Posted (April 7th, 2011 at 8:14 am PST) by Justin@DH

The gambino webserver has an issue with its primary hard drive, which has the operating system but no customer data. Our data center team will replace the drive and re-image it with the version of Debian linux we normally run, and after some configs it will be back up. Customer data is on a completely different raid array and is not affected, so no data should be lost nor are any restores necessary. When we have an eta on when this issue will be completed we will update this post.

Update Thu Apr 7 11:10:00 PDT 2011: The hardware swap and OS install is done, the last configs are running now and sites will start coming up. Sorry again for the issues with this server, but with this maintenance done it should be more reliable than previous.

Update Thu Apr 7 13:09:00 PDT 2011: The configs are all double checking its setup now, but there is traffic and it looks to be working fine now, so we believe this is resolved. If you notice any issues with sites or services not working, please contact support.


3:49 am

Outgoing Email Downtime (RESOLVED)

Posted (April 7th, 2011 at 3:49 am PST) by patrickm

We have identified an issue with sending email with SMTP from our servers at this time.  Our Admins are already working to resolve this issue and we will post any updates we have here.  Please check back for updates, or submit a Support ticket and we will notify you when this issue has been resolved!  Our apologies for the inconvenience and we thank you for your patience!

Update 4:52am PST: This issue is also affecting incoming mail for some customers, as well as WebMail access to email accounts. We are still working to get this resolved and will provide an update as soon as possible!

Update 5:40am PST: This issue is unfortunately ongoing. We believe we have identified what initially caused this problem, but getting it completely resolved is taking some time.  We will, of course, continue to update this post with any information we have.

Update 6:40am PST: The admins working on the issue have given us the all clear, that the problem has been resolved. Testing many accounts has shown no further errors, but if you still see issues please contact support.


5:10 pm

Shared server ‘ibanez’ being problematic.

Posted (April 3rd, 2011 at 5:10 pm PST) by Jason C

Customers on shared server ibanez are no doubt noticing issues reaching their server at this moment. Our admin team is currently having a heck of a time keeping the server up and stable.  Between BIOS errors, file system checks, losing the VGA port on console and freezing during boot, things are just not happy.  We’re sending someone over to the machine itself to give it all the TLC that it needs.

Please keep an eye here for updates as this issue progresses.

UPDATE 8:20pm PST – The machine came back online about 1.5hrs ago, but we’re keeping the post unresolved while running all the necessary updates and checks. Thank you for your patience. Check back soon for more updates. If you have a ticket in our support queue, we’ll get back to you as soon as we made sure everything is resolved.

UPDATE 9pm PST – All the updates and checks are now complete on Ibanez. If you’re still experiencing issues with your sites, please contact support with the details.


1:23 pm

Proposal: Disabling comment notification on EASY MODE WordPress and PhpBB

Posted (April 1st, 2011 at 1:23 pm PST) by brians

Again, this is just a proposal. We are considering disabling the ” comment notification ” feature on EASY MODE WordPress and PhpBB that were installed through our Panel. This applies to ALL current and future WP / PhpBB installs. Also, this WILL NOT have any impact on installs that were done in ADVANCED MODE or manually. This ONLY impacts installs that were done in EASY MODE.

You may ask why we are considering such action. Well, mail originating from our oneclick machines are being blocked by major providers ( yahoo, gmail, etc ). We feel this is due to users receiving comment spam from their WP and PhpBB installs. The notifications are arriving at their yahoo, gmail addresses. They realize that its spam and click ” YES, this is SPAM “. Low and behold, our machines are added to their block list.

This PROPOSED change is not as bad as it may seem. If you really need this feature, you can simply reenable through the softwares admin interface. However, we would just like to get your thoughts before we decide to make any changes. Speak Now or forever hold your peace!!!

UPDATE 2011-04-04 9:31AM PST

We’ve decided that the best course of action will be to disable ” comment notification ” by default. Again, this only impacts installs that were done in easy mode. Also, you will have the option to reenable the feature if you chose. We will update the post once the feature has been disabled. Please check back for updates.


7:06 am

shared server “apia” having hardware issues

Posted (April 1st, 2011 at 7:06 am PST) by robert

The server “apia” is having issues with its hardware. One of our admins is currently on their way to the datacenter to move the harddrives to a new chassis. We will update this post once we have any further information or the issue has been resolved. We apologize for any inconvenience this causes you.

Update – 08:05am (PST): This issue has been resolved. If you’re on this server and continue to have problems, please contact support and they will investigate it further for you.


10:43 pm

shared server ‘moran’ downtime for file-system check (RESOLVED)

Posted (March 31st, 2011 at 10:43 pm PST) by patrickm

We regret to report that the shared server ‘moran’ is down for a file-system integrity check which was required upon rebooting the server.  We do not have an ETA at this time, but we will post updates about this issue here.  Please contact support with questions!

UPDATE: The server is now back on-line and the drives are fine. Sorry for the inconvenience. If you have any problems/questions please feel free to contact our support department.


9:28 am

VPS host machine (homie-vserver11) down for file system check.

Posted (March 30th, 2011 at 9:28 am PST) by Brandon

One of our host machines was displaying some possible file system issues and we’ve taken it down for a file system check. Typically this takes several hours to complete and any VPS guests running on this host machine will not be accessible until it has finished.

This list includes all affected VPS guests:

ps10676 ps10746 ps10888 ps11350 ps11628 ps11968 ps13207 ps18605 ps25805 ps26053 ps26059 ps26090 ps26358 ps26555 ps26926 ps27511 ps42821 ps42836 ps43622 ps43821 ps46234 ps46243 ps4625 ps46268 ps4627 ps46417 ps4650 ps4652 ps4654 ps46556 ps4664 ps4667 ps46691 ps4673 ps4676 ps46843 ps4814 ps4823 ps4830 ps4888 ps4897 ps4903 ps4905 ps4993 ps5049 ps5054 ps5059 ps5088 ps5093 ps5222 ps6158

We sincerely apologize for any inconvenience this has caused and appreciate your continued patience.  We will update this post as soon as we have further information.

 

Update 12:38 PM PST: It looks like the FSCK ran into some problems along the way and had to be restarted. Unfortunately this means we’re now looking at 2-4 hours from this point forward until the VPS guests are back up and running.

 

Update 2:47 PM PST: The FSCK has completed and all VPS guests on this host machine are now back online. If you’re still having any issues related to this please contact our helpful tech support staff.


8:53 am

funchal server having RAID issues

Posted (March 30th, 2011 at 8:53 am PST) by danwy

The ‘funchal’ server is having issues with its RAID card. One of our admins is currently on their way to the datacenter to replace the faulty hardware and will have it back up shortly. We will update this post once we have any further information or the issue has been resolved. We apologize for any inconvenience this causes you.

Update – 10:30am (PST): This issue has been resolved. Our admin was able to swap out the faulty hardware and the server has been up for about the last 30 minutes. If you’re on this server and continue to have problems, please contact support and they will investigate it further for you.


6:15 pm

Dedicated Server Network Issues

Posted (March 29th, 2011 at 6:15 pm PST) by Justin@DH

We are experiencing a network outage in one of the racks at our LAX datacenter which contains dedicated servers. Our admin team is on the case and looking for the cause of the loss of connectivity. This is affecting a subset of dedicated servers only. When we have further info on this issue we will update this post. Our apologies for the network issues affecting you.

Update @ 7:21pm PST: As it stands, our admins have ruled out our networking equipment as the cause of the issue. We are continuing to dig into this.

Update @ 7:46pm PST: It would appear that due to an operator error the default routes were lost on the machines. This means that they couldn’t communicate to the outside world. Our team is pushing an update to the network configuration for all of the affected servers that should resolve this. Hopefully we’ll have an update for you in a matter of minutes.

Update @ 8:18pm PST: The update is taking longer than expected, but some servers should already be back online. If yours isn’t, it should be soon. There’ll be an update here once this is all taken care of.

Update @ 8:50pm PST: The update has finished, but there are still a few servers that need some personal attention from our admins. Most customers should be back up now tho. One final update will appear here once everything is cool.

Update @ 9:30pm PST: We’ve received an “all clear” from our admin team. The last few stragglers have now be reconfigured and a quick core router fix at our LAX facility was issued and everyone should be okay now. Any customers on a dedicated server still seeing issues are encouraged to write to support for assistance.


 
© 1996-2011, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
2:36 pm

Shared server ‘riddler’ running with degraded RAID performance.

Posted (March 29th, 2011 at 2:36 pm PST) by Jason C

After having a brief issue with the card that controls its RAID array and one of its drives, shared server riddler is back up and running — albeit with degraded drive performance.

As it stands, our admin team needs to replace a bad drive before the RAID can start rebuilding. That should happen shortly, but until then users on riddler may see some slowness on their sites.

Please know that we are doing everything we can to resolve this issue quickly. Once the server is back to 100%, we will post an update to this entry — so keep an eye on this entry if your sites are on riddler.

Update @ 2:59pm PST: Our admins are putting the new drive into the machine now. Until the RAID is completely rebuilt, there will still be degraded performance on the server. Folks with dynamic sites will be most affected by this, but hopefully the rebuild shouldn’t take too long to complete.

Update @ 5:35pm PST: Everyone on riddler should now be seeing an acceptable amount of performance from their sites. We’ve spot checked both dynamic and static sites on the server and they’re coming up rather quickly. If you happen to have any further issues and are on riddler, please don’t hesitate to contact our support team.


2:41 am

shared server ‘banjul’ networking issues

Posted (March 29th, 2011 at 2:41 am PST) by patrickm

The shared server ‘banjul’ is experiencing networking issues at this time.  Our techs are working to resolve the issue and we will post again here when it has been resolved!

Update 3:39am PST: This issue has been resolved!  Please contact support if you are having any trouble!


1:18 am

Moran down with RAID issues

Posted (March 28th, 2011 at 1:18 am PST) by JeffH

One of the drives in the server, Moran appears to have died, and is causing problems for the rest of the array.  We are working on getting it back up and running.  Will update this post when it’s back up or if a failover to new hardware is required.  We apologize for any inconvenience.

Update:  The drive array is unrecoverable, and we will be failing over to different hardware and restoring from backups.  Will update once this is complete.

Update: Moran is back up and currently restoring from backups.  We will post another update once the restore process is complete.  Be sure to contact our friendly support staff if you have any questions!

Update 2011-04-04 08:05 AM PST: The data restore is done. If you’re still having any issues with your web service on this server, please don’t hesitate to contact support.


3:44 pm

Network maintenance window: March 27, 2011 @ 10PM PDT

Posted (March 27th, 2011 at 3:44 pm PST) by Jason C

So we’ve got a new time and date for last week’s canceled maintenance window. Tonight (March 27th, 2011) at 10PM PDT our admin team will be working on replacing the flaky core supervisor with a new unit from Cisco.  We are hopeful that this will help resolve our recent networking issues.

Please note that while we are doing what we can to minimize the downtime that our customers see, there may be some spotty performance for users around this time.  Keep an eye on this status post for updates as the maintenance progresses.

 

–UPDATE– 2:53 PDT

This work is now complete, and the network is now stable and operating well. If you continue to have trouble accessing your site(s) at this time, please submit a support request so our staff can investigate your specific situation.


11:19 am

Downtime on homie-vserver179

Posted (March 25th, 2011 at 11:19 am PST) by brians

The machine is experiencing file system issues. You have likely noticed that your private server is not accessible. Our admins are currently investigating the matter. Unfortunately, we do not have an ETA at the moment. We sincerely apologize for any inconvenience this has caused and appreciate your continued patience. Below is a list of machines impacted by this downtime. Please check back for future updates.

ps27054 ps27057 ps27058 ps27059 ps27061 ps27062 ps27066 ps27068 ps27069 ps27070 ps27071 ps27073 ps27074 ps27075 ps27077 ps27078 ps27079 ps27094 ps27096 ps27097 ps27098 ps27099 ps27100 ps27101 ps27102 ps27104 ps27106 ps27109 ps27111 ps27113 ps27115 ps27117 ps27122 ps27123 ps27955 ps31189 ps35810 ps36888 ps37055 ps37457 ps37765 ps37777 ps37841 ps39442 ps39444 ps43881 ps46191 ps46422 ps46483

UPDATE: 2011-03-25 1:32PM PST

Sorry for the delay in posting the updates. Attempts to bring the machine online have been unsuccessful. The filesystem issues were resolved. However, a hardware failure surfaced. We had to re-image the machine to fix the hardware issue. At the moment there appears to be a network issue on the machine. Our admins are currently addressing the network problem. Sorry again for the inconvenience this has caused. Your continued patience is greatly appreciated. Please check back for future updates.

UPDATE: 2011-03-25 3:41PM PST

The machine is back up and running, but we need to manually restore some lost data (not customer data), which is going to take a few hours. Again, sorry for the inconvenience.

UPDATE: 2011-03-25 6:10PM PST

The server should now be back up and running. All guests should be up and running as well, so if you’re seeing a problem with your guest, please contact support right away!


12:05 am

shared server ‘kramer’ downtime due to RAID card replacement

Posted (March 25th, 2011 at 12:05 am PST) by patrickm

We are sad to report that the RAID controller in the shared server ‘kramer’ has failed.  One of our engineers is working to replace this card right now and the server should be back up and running shortly!  We will update this post when the work has been completed, thanks for your patience!

Update 2:40am PST: Apologies for the lack of update on this, as the server has been back up and running for several hours now.  If you are experiencing any issues from this, please contact Support.  Thanks!


3:45 am

VPS MySQL host homie-vsql55 downtime

Posted (March 24th, 2011 at 3:45 am PST) by patrickm

The VPS MySQL host homie-vsql55 is currently experiencing issues and we have dispatched an engineer to the data center to resolve the problem.  This is causing downtime for the following MySQL VPSs:

psmysql28860 psmysql28866 psmysql28899 psmysql28908 psmysql28933 psmysql28972 psmysql28997 psmysql29068 psmysql29134 psmysql29177 psmysql29221 psmysql29293 psmysql29660 psmysql29806 psmysql30083 psmysql30086 psmysql30260 psmysql30309 psmysql30425 psmysql30658 psmysql30736 psmysql30761 psmysql33732 psmysql34240 psmysql34448 psmysql34501 psmysql34514 psmysql35260 psmysql35278 psmysql35488 psmysql38591 psmysql38728 psmysql39567 psmysql39820 psmysql40041 psmysql40067 psmysql40503 psmysql40599

We will update this post when we have more information, or when the issue has been resolved!

Update 5:58am PST: We have been working on this and found that the RAID array is not working.  We are working on replacing the drives for this server and then we will be restoring from the most recent backups.  More updates coming!

Update 8:07am PST: All guests have been reset and our database expert is preparing the final restoration of data.

Update 9:05am PST: Our database expert has all restores underway and has estimated that it should complete within the hour for most databases (if any are particularly large it may take over an hour in those specific instances). We’ll update this post again once they are all completely restored.

Update 9:55am PST: Our admin team has confirmed that the restore is completed and the tests that they have run have come back clean so it looks like this issue is resolved. If you are on the affected host and are still experiencing issues please contact support for assistance.


8:52 am

Network maintenance window 2011-03-23, 10pm PDT – cancelled

Posted (March 23rd, 2011 at 8:52 am PST) by danwy

We will be performing some network maintenance tonight to replace the flaky core supervisor mentioned in previous posts. We just received a replacement from Cisco and are hoping that this will resolve the recent networking issues that have been posted about in detail on this status page. We are doing all that we can to minimize downtime during this maintenance, but you may notice some short outages around this time. Please keep an eye on this status post, as we will be updating it with information on how the maintenance is progressing.

UPDATE 23 Mar: Due to replacement hardware having issues, the maintenance window has been cancelled. We will post a new time when replacement hardware for the replacement hardware gets here.


10:25 pm

Network Outage

Posted (March 22nd, 2011 at 10:25 pm PST) by Sandon

We appear to be having a problem that is causing a network-wide outage. More updates to come as they are available.

Update: 10:34pm PST - Thanks for your patience.  The outage appears to have been the result of a form of DoS.  It has since been resolved, and we should be operating at full capacity.  If you are still experiencing issues, please contact technical support.

Update 11:40pm PST: Apologies, as this issue is not actually resolved.  We will update this post with more info as soon as possible!

Update 11:49pm PST: This issue is mostly affecting MySQL connections, but is also causing delays for mail delivery.

Update 3:57am PST: First, let me apologize for the lack of updates on this issue!  We have been working all night to resolve this problem and we have a fix in place.  The MCEC between our VSS switches was the cause for this issue.  For now, we have shut down one of the switches and are operating with degraded capacity on the affected rows until we are able to resolve the root problem.

We are awaiting a service call from Cisco, who will be helping us identify and resolve this problem.  Unfortunately, we will need to bring the MCEC back online at that time, which may cause more downtime.  We will be updating this post if and when that happens today.

Thanks for your patience through this difficult situation!  If you continue to have any issues with MySQL connections or email access, please contact Support and we will be more than happy to investigate!

Update 4:27am PST: We are re-activating this switch for Cisco troubleshooting to take place.  This may cause downtime and we will update this post as soon as it is completed.

Update 5:07am PST: We have had this switch enabled for the last 30 minutes and are seeing no issues, so this problem appears to be resolved for now.  We are planning to completely replace this switch supervisor and we will make a new post with details on when that maintenance window will occur.  Thanks again for your patience and contact Support with any issues!


6:50 am

godin shared webserver down – resolved

Posted (March 21st, 2011 at 6:50 am PST) by douglas

godin is having hardware issues (motherboard is suspect). We’re in the process of failing the machine over to new hardware.

Update 7:29am: The hard drives are working fine in their new hardware without any issues or a need for an fsck. “And there was much rejoicing.” — Monty Python.

Update 8:03am: We had to replace the RAID card due to the infamous “arcmsr0: abort device command” issue we’ve been seeing. Fortunately the the raid card manufacturer has identified a potential issue with their cards and are working on a solution for us. “And there was much rejoicing.” Yay!


 
© 1996-2011, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
2:00 am

Network maintenance window 2011-03-19, 10pm PDT – 2am PDT

Posted (March 20th, 2011 at 2:00 am PST) by Brian@dreamhost

We have a scheduled network maintenance window coming up, Friday March 18th form 10PM PDT to the next morning, at 2am PDT. This window will update a few different issues on our network, and hopefully clean up any lingering interface issues we had from Monday’s outage.

Our network engineer has pledged to do everything he can to make sure this update is as transparent to our customer as possible. We only anticipate some slowness, or occasional disconnects, and don’t expect any downtime. Being prepared, however, this is always the possibility of downtime.

We apologize for this, and hope to make this update as seamless as possible!

March 19th Update 10:25 PM PST: Our apologize this was actually postponed until tonight and the status post was not updated. No work/maintenance was done yesterday.


11:16 pm

VPS host down for fsck

Posted (March 19th, 2011 at 11:16 pm PST) by Sandon

One of our vserver hosts was showing some issues with its file-system so we had to bring it down for a fsck. This process can take 2-5 hours depending on the amount of data but its usually quicker on our VPS hosts as they don’t have as much data. This is affecting the following VPS guests:

prelude-relay-lax1 ps23350 ps23521 ps23593 ps23671 ps23710 ps23742 ps23781 ps23831 ps24069 ps24124 ps24166 ps24298 ps24377 ps24537 ps24745 ps24822 ps25140 ps25169 ps25440 ps25595 ps25663 ps25721 ps25801 ps26099 ps26615 ps26618 ps26619 ps27272 ps27585 ps27629 ps27635 ps27900 ps27969 ps27987 ps28028 ps29012 ps29207 ps29355 ps34378 ps35860 ps36649 ps36769 ps36809 ps36812 ps36824 ps36855 ps36944 ps36977 ps37152 ps37947 ps39236 ps41776 ps42330 ps423566 ps42574 ps43161 ps43220 ps43403 ps43416 ps43768 ps43770 ps43942 ps44004 ps44026 ps44161 ps44162 ps44201 ps45048 ps46486 ps47085 ps47091 ps47102 pss47137

Update 9:45 AM PST: The filesystem check is done and all of the VPS guests listed above are up and running. If you are having any further issues with your sites being down, please contact our support team.


12:41 pm

Hosting Machine Silversurfer Down Due to Hardware Issues

Posted (March 18th, 2011 at 12:41 pm PST) by mir

The admin team is working to fix a hardware issue with the Silversurfer server as there were issues with the backup hardware we were trying to move it over to as well (normally it would have been back up already). They have estimated an hour until it is returned to service so we will update this post when we have more information (or upon completion).

Update: 2011-03-11  3:30 PM PST: We are in the process of restoring all data from backups – in the meantime the sites on this host are running directly off of the backups as the restoration runs and should be available (but at a reduced performance level until that completes). We’ll update this post again when that finishes up.

Update: 2011-03-11 9:01 PM PST: We were able to fix the “old” silversurfer so there will be a brief downtime now and silversurfer will be back to its old self. Once it is back up and running any change you have made today while on the “new” silversurfer will be rsynced back to the “old” silversurfer so that no data will be lost. There will be an update here once the “old” silversurfer is back up and surfing.

Update: 2011-03-11 11:01 PM PST: For the last 20-30 minutes services has been back up on the original array so all data minus stuff uploaded today should be restored. I am now restoring the new data that was uploaded since then and will update this status post when that is complete.

Final Update: 2011-03-11 04:00 PM PST: The restore is done. If you are still experiencing issues with this server, please contact support.

Update: 2011-03-13 06:20 AM PST: The boot partition appears to have some corruption from the previous raid issues now that its back on the original array. The /home partition where user data resides seems fine but the OS must be reinstalled. ETA 1 hour before the server is back up.

Update: 2011-03-13 07:30 AM PST: It looks like the home partition is also showing some issues so an fsck is running on that. ETA is 2-5 hours.

Update: 2011-03-13  3:00 PM PST: Unfortunately, the file system check is taking longer than expected and is still running. We’ll update this post again as soon as it’s done and iwith any other progress. We appreciate your continued patience.

Update: 2011-03-13  7:40 PM PST: I’m happy to report that the problems with your web server, Silversurfer, are now resolved. The file system check completed, and all sites I randomly spot checked are loading fine. If you’re still having issues with your domains, please submit a ticket with the specifics, so we can take a closer look.

Update 2011-03-18 8:54 PM PST: Silversurfer locked up and on reboot decided to fsck the home partition again. Repeatedly needing to have a fsck ran is a sign of a serious problem with the file system so we are going to play it safe and go ahead and move silver surfer to new hardware. We will update this post once it is up and running again.

Update 2011-03-19 3:00 AM PST: There will be some slowness for a couple days but everything is back up and running on the new hardware/array.

Update 2011-03-28 13:00 PM PST: The data restore is done. If you’re still having any issues with your web service on this server, please don’t hesitate to contact support.


10:00 am

PHPMyAdmin broken.

Posted (March 18th, 2011 at 10:00 am PST) by mir

Some of our database servers were having issues with the PHPMyAdmin setup which was resulting in an inability to access that feature. Fortunately this did not have any impact on actual database availability or performance and we fixed most of them fairly rapidly but there are still a few machines which are affected:

barbaro, catshark, bennett, cykotix, domination, aslan, houndshark, drooper, hazelnut, indiana|, joecool, ein, jingoro, misha, petrelli, putney, sandtiger, scooby, ragland, sprockle

We should have those fixed up in just a few minutes as well. We’ll update this post when that is finished or we have further information that needs to be posted.

Update 10:05 PST: The admin team has finished fixing this issue and ran through and double-checked to be certain and all looks well. The cause appears to be an error in our package update system which we’re investigating to ensure this does not occur again.


1:39 am

Server “tobias” is offline for a RAID replacement

Posted (March 17th, 2011 at 1:39 am PST) by RLP

We have had to take the server “tobias” offline for  a while to accomplish a RAID replacement. We hope to have this completed, and the server back online, shortly.  We have no firm estimate, however, of how long this will take but we expect it to take around 15 minutes. We will update this post as more information becomes available. I am sorry for the inconvenience this causes, and we are doing all we can to get the server back into service quickly.

–UPDATE — 01:51  PDT

The raid card in “tobias” has been replaced in good order, and the machine is now back in full service. I am sorry for any inconvenience this has caused. If you are having trouble with this server at this time, please contact support for further assistance.


4:54 am

Server “rangoon” is offline for a RAID replacement

Posted (March 16th, 2011 at 4:54 am PST) by RLP

We have had to take the server “rangoon” offline for  a while to accomplish a RAID replacement. We hope to have this completed, and the server back online, shortly.  We have no firm estimate, however, of how long this will take. We will update this post as more information becomes available. I am sorry for the inconvenience this causes, and we are doing all we can to get the server back into service quickly.

–UPDATE — 05:47 PDT

The raid card in “rangoon” has been replaced in good order, and the machine is now back in full service. I am sorry for any inconvenience this has caused. If you are having trouble with this server at this time, please contact support for further assistance.


2:20 am

Network outage -RESOLVED-

Posted (March 14th, 2011 at 2:20 am PST) by patrickm

We apologize for the downtime as our core network is offline at the moment. Our network admin is addressing the issue and we expect service to resume shortly. This will prevent access to the customer web panel.

Update 4:00 am (Pacific): This issue is now resolved!  This was due to network maintenance which was not supposed to have any effect on connectivity, but sadly did.  We are investigating this more thoroughly and if we have any additional info to provide, it will be posted here!  Thanks for  your patience with this outage.  Please contact Support if you are still experiencing any issues.

Update 5:41 am (Pacific): We apologize, as we are experiencing another outage.  Our network admin is working on the issue at this time and we expect it to be resolved shortly.  This outage will affect core services such as the Web Panel and WebMail.  We will update this post as soon as we have more information or when the issue is resolved!

Update 7:11 am (Pacific): We are currently operating on half of a core, so partial connectivity has been restored. We are working to resolve an issue with one of the cisco core supervisors not maintaining it’s configuration. Systems that are affected by this are our servers located at our downtown Los Angeles, CA location (including the panel)– servers at our LAX data center are not affected by this outage.

Update 11:15 am (Pacific):Connectivity has been restored to a large part of our network. We are still experiencing routing issues that may cause outages for certain machines located in our LAX or Downtown Los Angeles data centers. This post will be updated as soon as we have more information on the source of these outages.

Update 1:45 pm (Pacific):We’re sorry for the lack of updates on this issue. Unfortunately, there haven’t been any updates here because we don’t have any new information. Our network admins have been working non-stop since this morning to fix these problems, but we’re still not sure when it will be resolved. This post is just meant to let you know that we are doing everything we possibly can to restore our system to normal and that we are not ignoring the situation. If you have already submitted a support ticket regarding this, you will receive updates by email as soon as we have them. Again, we apologize for the inconvenience this has caused you and thank you for your patience as we try to get things back to normal.

Update 3:45 pm (Pacific):We have just received word from our admin team that these issues have been resolved now. All services should be restored at this point to (web)mail, mysql and web servers. As we posted above, one of our main switches in one of our datacenters locked up and needed to be rebooted. When it came up, it was missing its configuration files. This caused all kinds of routing issues on our backend and resulted in many vital servers to lose their connections. Unfortunately, there was no way to recover the configs, so they had to be rebuilt from scratch. Most of the day was spent reconfiguring the switch, so we didn’t have any new updates other than what was already posted above. We’re currently going over all the information we have and will have much to say about it over the next few days. Please keep an eye on this status page, as well as our blog (blog.dreamhost.com), for more specific details on what caused this outage and what we plan to do to prevent it in the future. Again, we can’t apologize enough for the inconvenience this has caused our customers. Please contact us with any further questions/comments you have and we’ll be happy to assist you.

Update: 5:30 pm (Pacific): Our admin team has noticed a bit of internal packet loss that seems to be related to the earlier networking issue. This is causing some issues with sites that connect to MySQL databases. So while we’re not out of the woods just yet, we are aware of the problem and are working on fixing it up now.

Update: 6:20 pm (Pacific): Even tho the admins are uncovering more configuration issues, things are getting better for a number of folks. Customers who were still seeing slowdowns with POP, IMAP and webmail may notice improvements after we tweaked a setting. Other folks with MySQL issues may be seeing a speed increase as well. We’re still pounding away at this, so don’t think we’ve called it a day if you’re still seeing issues. Hopefully we’ll have more updates soon!

Update: 7:28pm (Pacific):The network is confirmedly looking much better, but we continue keeping an eye on it. If you’re still having any issues, please submit a ticket with as much details as you can provide, so we can use that to evaluate if your case is unrelated or if there is still something else lurking around waiting to be fixed. We’ll update again as the evening progresses, but so far so good. We very much appreciate your patience and understanding regarding this outage.

Update: 3/15 – 1:00pm (Pacific): We’ve been keeping an eye on things over night and can confirm that we have not seen any further issues relating to this outage. If you have any further issues, please contact support for further assistance. We thank you again for your patience as we worked on resolving this problem.


8:40 am

Phoenix Down (hardware)

Posted (March 12th, 2011 at 8:40 am PST) by mir

The server Phoenix has a bad RAID card so our admin team is rushing to replace it. The current estimate is around 35-45 minutes and we’ll be sure to update this post as soon as we have further details or when the issue is resolved.

Update 09:30 PST: Our admin team has replaced the RAID card and all services are back up and running!


3:49 pm

Network maintenance tonight, 2011-03-11, 10pm PST.

Posted (March 11th, 2011 at 3:49 pm PST) by Brian@dreamhost

We’re planning on updating a few of our network routes tonight, but do not anticipate this causing any downtime at all. You may see some slowness during this time, however.

We will be performing this update between 10pm through 2am, PST.

Update 4:50 AM PST: Other than a short period where we had problems (20 minutes) everything seemed to be working properly until later we noticed that a small number of our subnets were showing a high latency spike and around 30% packetloss caused by high cpu-usage on our new router at One-Wilshire. We have since shutdown our transit on that router and the problem appears to be resolved and everything back to normal.


3:46 am

Shared server ‘aquarius’ going down for hardware maintenance

Posted (March 11th, 2011 at 3:46 am PST) by patrickm

We have found an issue with the RAID controller on the ‘aquarius’ shared server and it is being replaced. This will be starting around 45 minutes from now (4:30am Pacific Time). The maintenance should only take a few minutes and then the server will be back up and running. Please feel free to contact support with any questions and we will update this post when the maintenance is complete!


 
© 1996-2011, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
1:24 pm

Removing VanillaForums and OpenAtrium from our One-Click Installs

Posted (March 10th, 2011 at 1:24 pm PST) by andrea

Vanilla Forum and OpenAtrium are being removed from our one-click installer for the time being.

If you are currently using either of those installs on your DreamHost domains, the installs will remain untouched, but you won’t be able to upgrade them through the panel any more; they will need to be upgraded manually. Please contact the developers for further instructions if you need help with those upgrades, as our support team won’t be able to assist you.

We are likely going to bring them back, but we don’t have a timeline yet. If you have any questions regarding this matter, please contact our support team through the panel under Support/Contact Support.


11:42 pm

Server Alvarez is offline for a RAID replacement

Posted (March 9th, 2011 at 11:42 pm PST) by RLP

We have had to take the server “alvarez” offline for  a while to accomplish a RAID replacement. We hope to have this completed, and the server back online, shortly.  We have no firm estimate, however, of how long this will take. We will update this post as more information becomes available. I am sorry for the inconvenience this causes, and we are doing all we can to get the server back into service quickly.

– Update 11:49 PM PST —

This replacement has been completed, and the server is now online and functioning properly. If you are experiencing problems with this server, please contact Support.


7:41 am

Speartooth Mysql Server – Raid Issues * RESOLVED *

Posted (March 9th, 2011 at 7:41 am PST) by Justin@DH

The mysql server named speartooth went unresponsive a short time ago. Our admins believe its RAID card has gone bad, as it will not boot with the raid array. We are investigating now and will likely need to replace the raid card. When we have more info or an ETA, we will update this post. Our apologies for the downtime related to this hardware issue.

— Update 8:04AM PST —
The issue has been resolved, the raid card behaved unusually and after removing a failed disk from the raid array, stopped recognizing the raid volume. The bad disk has been removed, and speartooth is running now. It has to run a large number of “repair table”‘s to fix the open tables when it went down, so it will be slow for possibly up to 15 minutes while this completes. When it is done repairing and back to normal, I’ll update this post one last time.

— Update 8:30AM PST —
The repair of such a huge number of databases at one time was too much for speartooth to take without possibly going down. We are slowly starting each of the 50 services on it, letting the tables have time to repair. Full service should be restored and back to normal within 15 minutes we are expecting.

— Update 9:07AM PST —
Despite signs of improvement, speartooth again has went down due to raid issues. We are evaluating the situation and the fastest solution, if it should get new hardware or if it should be moved to other mysql servers in the same rack. We will update this post when we have more information and an ETA.

— Update 9:30AM PST —
An admin is on their way to replace the raid card in the server. If this fixes the severe issue it is having, service will be restored. If it appears the raid volume is not recoverable, we will be replacing the hardware. More info to come. Sorry again for the issues with this server, raid issues are very rare on our mysql servers and this is not a typical outage.

— Update 10:58AM PST —
The raid card replacement got the server working again, but the raid array has failed. We will be reinstalling the debian OS on it, and will have to restore the databases from backups performed between 1 and 3AM this morning, which was a couple hours before the hardware issues started happening. I will update this post again when the server is live again, and when restores of backups begins and an ETA on its completion.

— Update 11:23AM PST —
Operating system is installed, services being setup on it now, and restores will begin in likely 15 minutes.

— Update 12:24PM PST —
Databases are half restored, we have multiple channels of restores going to get these all back in service as soon as possible. As soon as one of the channels finish, we’ll turn that service on and it will be live. When the restore is completely done I’ll do a final post.

— Update 1:43PM PST —
We have around 90% of the dbs restored, the last few services are being restored still. We should have full service restored for all customers within the next hour.

— Update 3:00PM PST —
The services zehner, zombo and zukowski are the last to restore and are taking a bit longer than expected due to larger databases, the other services are restored and live. When these last 3 services are finished, we will do a final check and resolve this status post.

— Update 3:43PM PST —
The last restores have finished, regrants are run, and all should be live again. If you are still having any issues, please contact support.


9:41 pm

Networking issues causing intermittent connectivity for shared and VPS customers.

Posted (March 8th, 2011 at 9:41 pm PST) by Jason C

Some customers are reporting issues where their sites have become slow or unresponsive for a brief period of time. There are even some less fortunate folks who are seeing repeated flaps like this on their sites and services.

Our admin team is currently looking into what might be causing this now. We will update this post as soon as we have more information to share.

Update: This issue has been resolved by our Network team!  This was due to a bad link in one of our data centers. Once we have more details on this issue, it will be provided! Thanks for your patience. If you still experience any issues, please contact Support!


4:38 pm

Damascus server RAID issues

Posted (March 8th, 2011 at 4:38 pm PST) by danwy

The damascus server has been having some issues with its RAID card over the last hour or so, causing sites to be intermittently down. One of our admins is currently on his way to the datacenter to replace the faulty hardware. Service on it for the next hour may be spotty, but after that things should be running smoothly again. We’ll update this post again once the RAID card swap is complete. We’re sorry for any inconvenience this causes you.

Update: The RAID card was successfully replaced a short time ago and our admin team has been monitoring the new hardware since then to make sure everything is cool.  According to them, everything is holding steady at this time. If you happen to notice any issues with your sites (and they are hosted on damascus) please don’t hesitate to contact support ASAP.


11:21 am

Homie-vserver158 going down for a RAID card replacement.

Posted (March 3rd, 2011 at 11:21 am PST) by Brandon

Stability issues on the VPS host machine “homie-vserver158″ have been tracked down to a faulty RAID card. It is currently being replaced, and the server should be back online with 15-30 minutes.

This issue affects the follow VPS guests:

ps23363 ps23387 ps23460 ps23516 ps23667 ps23737 ps23826 ps23862 ps24475 ps24476 ps24478 ps24482 ps24548 ps24565 ps24567 ps24599 ps24605 ps24669 ps24739 ps24988 ps25019 ps25032 ps25150 ps25248 ps25321 ps25398 ps25416 ps25471 ps25496 ps25602 ps25671 ps25722 ps25966 ps25989 ps26137 ps26242 ps26469 ps26626 ps26845 ps27433 ps27554 ps28181 ps28183 ps28298 ps29029 ps29385 ps29424 ps29924 ps37690 ps37746 ps42437 ps42470 ps42484 ps42501 ps42538 ps43124 ps43172 ps43268 ps43270 ps44099 ps44106

We will update this post when the issue has been resolved, or when there is updated information.

 

 

Update: 2011-03-03  11:37 AM PST:

The new RAID card has been installed and the machine is back online.  If you are still experiencing any issues, please contact Tech Support.


12:53 pm

homie-vserver129 is down due to network issue

Posted (March 2nd, 2011 at 12:53 pm PST) by brians

Our admins are currently investigating the matter. The Vserver guests listed below are affected. We do not expect downtime to last more than 30 minutes. We sincerely apologize for any inconvenience this has caused and appreciate your patience. Please check back for future updates.

ps19705 ps19713 ps19736 ps19961 ps20036 ps20097 ps20136 ps20409 ps20502 ps20625 ps20668 ps21645 ps21954 ps22076 ps22089 ps22111 ps22135 ps22736 ps22737 ps22843 ps23090 ps23103 ps23107 ps23326 ps23502 ps23571 ps23803 ps24019 ps24143 ps24185 ps24231 ps24904 ps25176 ps25375 ps25390 ps25396 ps25953 ps29070 ps29218 ps30236 ps30253 ps30956 ps31369 ps34839 ps36207 ps36766 ps36779 ps36965 ps37253 ps37324 ps37326 ps37333 ps37377 ps37719 ps37913 ps38030 ps38050 ps38085 ps38223 ps38745 ps38768 ps39314 ps39341 ps45476

UPDATE 2011-03-02 02:52:48 PM PST

Everything has been resolved. The Host machine is back online. There was a brief dns hiccup. Which made the vserver host inaccessible. Again, we sincerely apologize for the inconvenience this has caused. If you are still having problems, please contact Tech Support.


12:50 pm

jackson shared web server RAID card replacement

Posted (March 1st, 2011 at 12:50 pm PST) by douglas

jackson has shown a trend of raid card timeouts over the past week, so we’ll be replacing the card with a fresh one later today. The total time to replace the card is about 10-15 minutes.

Update: RAID card has been replaced.


12:22 pm

Email delays

Posted (March 1st, 2011 at 12:22 pm PST) by danwy

Mail is backed up on some of our relay machines, causing some mail delivery to be delayed. Our admin team is currently working on cleaning these queues up and getting mail flowing again. We’ll update this post when things are back to normal.

Update: 3/3 – 12:30pm: This issue should have been resolved now. While the mail queue is still high on these servers, they are now within normal levels. Keep in mind, these servers are the fallback servers that attempts to complete delivery of a message if the initial send fails. If you notice any emails being delayed still, please forward the headers from the message on to our support team and they’ll be happy to look into it for you.


12:20 pm

Vserver host ‘homie-vserver52′ downtime due to hardware issues

Posted (February 28th, 2011 at 12:20 pm PST) by jordan

The Vserver host ‘homie-vserver52′ is currently down for a reimage on new hardware. This affects the following VPS guests:

ps10005 ps10036 ps10043 ps10072 ps10250 ps10256 ps10258 ps10260 ps10266 ps10463 ps10465 ps10477 ps15771 ps15772 ps15773 ps15774 ps16167 ps20608 ps21820 ps21844 ps21862 ps25165 ps26648 ps27053 ps28078 ps28662 ps30185 ps30792 ps33547 ps40172 ps40203 ps40359 ps40557 ps40803 ps40837 ps40952 ps41032 ps41261 ps41357 ps41787 ps41853 ps41962 ps42654 ps43454 ps43581 ps43639 ps43931 ps44135 ps44171 ps44185 ps44186 ps9243 ps9256 ps9262 ps9263 ps9269 ps9282 ps9290 ps9295 ps9299 ps9303 ps9310 ps9315 ps9319 ps9327 ps9330 ps9337 ps9619 ps9633 ps9637 ps9641 ps9863 ps9873 ps9880 ps9931 ps9937 ps9946

We will update this post as soon as this process is complete, or if we have any updates in the meantime!

Update 2011-03-01 06:50 AM PST: We are in the process of restoring all data from backups – in the meantime the VPS guests on this host are running directly off of the backups as the restoration runs and should be available (but at a reduced performance level until that completes). We’ll update this post again when that finishes up.

Update 2011-03-01  16:25 AM PST: Unfortunately the initial restore we had running failed. We have restarted it and data is being restored again.  To clarify the previous update, all VPS servers on this host machine are up and working however, the data on the servers has not been restored. It will be restored shortly, but this is why you see empty home directories right now. We will update this once we have more information for you.

Update 2011-03-03  11:48 AM PST: The restore is still ongoing. There is a large amount of data which is why the restore process is slow, but some customers should have completely restored VPS’s  now. If your VPS is still missing data, please bear with us as the restore will get to your VPS soon.  We sincerely appreciate your patience while we get your data restored.

 


 
© 1996-2011, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
10:00 pm

Dreamhoststatus.com outtage * RESOLVED “

Posted (February 25th, 2011 at 10:00 pm PST) by jordan

Unfortunately, the server that houses dreamhoststatus.com had a bit of a problem tonight, and crashed. We have been working with our remote partners to restore this service (as dreamhoststatus.com is hosted off our normal network), however it looks like a complete hard drive failure has happened! We’ve lost all data for the time being.

We’re working to restore that, but for now, this blank domain we have here, will serve the same purpose. We’ll keep this site updated with further info on what happens.

Luckily, this only effects dreamhoststatus.com, and no customer sites/databases/anything else are effected!

UPDATE
2011-02-28 12:38 PST

The status page is back online. As you can probably tell. All previous posts have been restored as well. Again, we apologize for any inconvenience this has caused. If you have any additional questions, please contact tech support.


3:55 am

Shared server ernieball having raid issues *RESOLVED*

Posted (February 24th, 2011 at 3:55 am PST) by Ken

Shared server ernieball is having raid issues. Our admins are working to resolve the issue as quickly as possible. Please be patient while our admins dig into this.

UPDATE 05:03 Pacific: The raid card has been replaced and shared server ernieball is back online. If you continue to experience problems, please contact technical support through the normal channels.


1:29 am

shared server rabat lost its motherboard *RESOLVED*

Posted (February 24th, 2011 at 1:29 am PST) by Ken

Our admins are currently moving shared server rabat to new hardware. Please be patient while we work to resolve this issue.

UPDATE 02:58 Pacific: rabat has been moved to new hardware and is tentatively looking good for now. We will continue to monitor shared server rabat for a little bit to make sure that it does indeed stay up.

Thank you for your patience while we work through these issues.

UPDATE 05:06 Pacific: It looks like shared server rabat likes it’s new home. If you experience any problems, please contact technical support through the normal channels. Thank you for your patience while we work on improving your DreamHost experience.


8:51 am

homie-vserver165 getting new RAID card

Posted (February 23rd, 2011 at 8:51 am PST) by douglas

homie-vserver165 has been suffering from RAID timeout issues and will now be getting a brand new RAID card.

The card should be replaced on or before 9:30am PST.

Affected guests:
ps23353 ps23467 ps23640 ps23674 ps23712 ps23744 ps23783 ps23833 ps23914 ps23957 ps24126 ps24168 ps24842 ps24858 ps24929 ps25011 ps25355 ps25439 ps25528 ps25541 ps25719 ps25798 ps25816 ps25980 ps26298 ps27366 ps27745 ps28430 ps28437 ps28443 ps28453 ps28454 ps28538 ps28665 ps28678 ps29032 ps29224 ps29237 ps29634 ps29798 ps29801 ps29904 ps30034 ps30102 ps30250 ps30555 ps30649 ps31151 ps31204 ps33408 ps33451 ps33646 ps35337 ps35557 ps35837 ps35941 ps38168 ps38178 ps39342 ps41653 ps41680 ps41701 ps42328 ps42899 ps43135 ps43138 ps43148 ps44446 ps44451 ps44802 ps44847 ps45047 ps45062 ps45100 ps45106 ps45200 ps45260 ps45264

We’ve been having issues with cards we’re getting back from RMA returns as of late from a specific company. The new card is not one of those, so there shouldn’t be any further issues with this server.

Update 10:00a: The new raid card is in and vps guests are now starting up.


12:33 pm

Homie-vserver186 down and getting new RAID hardware

Posted (February 22nd, 2011 at 12:33 pm PST) by Leon

This affects the following customers VPS:

ps27249 ps27299 ps27343 ps27353 ps27359 ps27378 ps27398 ps27498 ps27499 ps27510 ps27558 ps27580 ps27588 ps27604 ps27627 ps27657 ps27736 ps27765 ps27799 ps27831 ps27940 ps27961 ps28067 ps28265 ps28310 ps28311 ps28554 ps28614 ps28616 ps28954 ps29173 ps29188 ps29238 ps29247 ps29316 ps29376 ps29418 ps29444 ps29455 ps29896 ps30652 ps30720 ps30898 ps31031 ps31144 ps31153 ps34274 ps35069 ps35193 ps35307 ps35790 ps40103 ps42101 ps42102 ps42139 ps42191 ps42252 ps42498 ps43113 ps43274 ps44012 ps44464 ps45016 ps45362 ps45374 ps45418 ps45425 ps45436

We have dispatched a technician to our datacenter to replace the faulty hardware and will have it up shortly. Thank you for your continued patience.

Update 2:05pm PST: The raid card has been replaced and the server is back up and running.  If you are having any issues with your sites, please contact our friendly support staff!


1:19 pm

Announce lists delayed

Posted (February 20th, 2011 at 1:19 pm PST) by andrea

We had some problems earlier with the scheduled lists not going out, but that has been fixed. However, we’re now looking at an unusually large mail queue for the actual pending messages to the individual subscribers on all those newsletters, so it may be a while for all those to arrive. We’re keeping an eye on it, and will update this post as soon as we make progress. Sorry about the inconvenience this is causing!

UPDATE 20:30 PST: All previously pending newsletters are now sent, and the mail server is continuing to send out the rest of the messages – the queue is significantly lower, back to the normal range and continues to remain so. Please contact support directly if you need more help with your announce lists, as it’s most likely unrelated to this problem.


7:10 am

Dedicated MySQL upgrade 5.0.x -> 5.1.x

Posted (February 19th, 2011 at 7:10 am PST) by Sandon

Next Tuesday (2/22) at 11 PM PST the following dedicated servers will be having a short downtime to upgrade mysql from 5.0 to 5.1. The actual downtime should be less than 10 minutes for each machine and all should be finished up by 1 AM. Any impact on customers would be those running php4 based sites which access mysql which do not support the new authentication method in mysql and they should be upgraded to php5.

Here is the list of servers:

ping reallifelab sauron sax searchmeister secrets settlements slaughter shinynew shodan suba sitepartner sunimage sunmachine tekhost timber telkoth toddmuchmore vantec webwerks vmachine yaleweb wpcf


6:07 pm

Mail moves are taking longer than expected

Posted (February 17th, 2011 at 6:07 pm PST) by andrea

If you have received a message from us in the last few days with the following subject and are currently experiencing login or send/receive issues with some or all of your email address, please know that we’re working on the problem.

Message by homie mail mover. (Message #xxxxxxxx)
Subject: Moved account #

We apologize for the inconvenience and will update this post as soon as we have news. You may also submit a ticket to support with the details and we’ll be happy to look into it for you, as it may or may not be related to this move.


3:43 am

Vserver host ‘homie-vserver179′ downtime due to RAID issues – RESOLVED

Posted (February 17th, 2011 at 3:43 am PST) by patrickm

We apologize for the inconvenience that this is causing.  The RAID controller in this Vserver host is being worked on at this time to improve stability until we are able to replace that hardware.  We will update this post as soon as more information is available.

Affected VPS guests:
ps27054 ps27057 ps27058 ps27059 ps27061 ps27062 ps27066 ps27068 ps27069 ps27070 ps27071 ps27073 ps27074 ps27075 ps27077 ps27078 ps27079 ps27094 ps27096 ps27097 ps27098 ps27099 ps27100 ps27101 ps27102 ps27104 ps27106 ps27109 ps27111 ps27113 ps27115 ps27117 ps27122 ps27123 ps27955 ps31189 ps35810 ps36888 ps37055 ps37457 ps37765 ps37777 ps37841 ps39442 ps39444 ps43881

Update 6:30am: We have replaced the RAID controller in this server and it is back up and running again!  If you continue to have any issues whatsoever, please contact Support and they will be happy to help!


3:33 am

shared server ‘cassiopeia’ downtime due to RAID issues – RESOLVED

Posted (February 17th, 2011 at 3:33 am PST) by patrickm

We are sorry to report that the shared server ‘cassiopeia’ has been experiencing intermittent downtime all night due to an issue with the RAID array.  Our engineers are working to resolve these issues and hope to have services returned to normal shortly!  We will update this post with more information when it becomes available.  Thanks for your patience!

Update 6:30am: We have replaced the RAID controller in this server and it is back up and running again!  If you continue to have any issues whatsoever, please contact Support and they will be happy to help!


 
© 1996-2011, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
3:35 pm

Shared server ‘lima’ hardware issues causing downtime

Posted (February 14th, 2011 at 3:35 pm PST) by jasony

We apologize for the downtime caused by this outage.  The shared server ‘lima’ is down due to hardware failure.  We will be moving the drives to a new chassis and will give an update once we have the server up and running.  If you have any questions, please contact our friendly support staff!

Update 4:10pm PST: lima is now back up and running!  If you are having any issues with your website, please contact our support staff!


12:04 pm

Shared server ‘minsk’ file system check downtime

Posted (February 14th, 2011 at 12:04 pm PST) by andrea

We are currently running a file system check on web server Minsk and trying to fix what’s causing its instability.
Very sorry about the downtime; we’ll update this post as soon as we have new information.

UPDATE 2:15pm PST: Unfortunately, it looks like the file system check will take some hours to complete, but we’re hopeful it will fix the issue that made it necessary to be ran in the first place.
Please check back here often for updates.

UPDATE 5:20pm PST: It looks like the file system check was restarted, but it’s estimated to finish again within a few hours. Sorry about the lack of good news this time, but please check back for more updates.

UPDATE Feb 15, 02:00 Pacific: The filesystem check is coming along – we are currently in a reboot cycle to verify that the filesystem has completed all checks. Thank you for your patience while we strive to improve your Dreamhost experience.

UPDATE 05:58 Pacific: The filesystem checks have completed and the reboot cycles have sustained valid reboots. Shared server minsk is looking good for now. We are keeping an eye on it for a little longer to verify that it has truly finished and is up. Thank you for your patience while we work on these hardware gotcha’s striving to improve your Dreamhost experience.

Update 3:45pm PST: Looks like one instance of apache is still down. We are working on fixing it. Sorry for the late notice!

Update 4:05pm PST: Now that’s fixed, too! If you’re still having trouble, please write a heartfelt e-letter to your best good support Dreamhost friend.


8:37 pm

Shared server ‘eskimos’ suffering from drive issues.

Posted (February 13th, 2011 at 8:37 pm PST) by Jason C

A short time ago, ‘eskimos’ started reporting problems with one of its drives. Our admins have already sprung into action, but we wanted to let you folks know that if you are seeing problems with your sites (and you happen to be on eskimos) we’re already aware that there’s an issue and are working towards a resolution.

More updates will follow just as soon as we have info to share.

UPDATE 2011-02-13 10:00 PM PST
We have resolved the issue with eskimos hard disk. This drive did not contain customer data, so sites should not be affected. If you are still experiencing issues please contact technical support.


8:00 pm

Firewall machines being upgraded the evening of February 11th.

Posted (February 11th, 2011 at 8:00 pm PST) by Jason C

Update: Done at 22:04 PM PST
Update: Beginning upgrade now.

The evening of February 11th, our admin team will be upgrading the core operating system on our internal firewall machines. We’ve been told that the method they’re using to do so should result in no downtime for our customers, but we still thought it was necessary to make you nice folks aware of the window in case something does happen to go awry.

We’ll update this entry as the upgrade happens.

Update: This will start at 10pm tonight and the window is from 10pm pacific time to Midnight.


7:47 pm

MySQL issue with homie-vsql62

Posted (February 11th, 2011 at 7:47 pm PST) by Oscar

Currently homie-vsql62 is down. The server halted during POST and it looks to be an issue with it’s hardware. Our Network Operations Center is addressing the issue and we expect to have the server back up shortly! Thanks for your patience and please contact Support with any questions you have.

Affected servers:

psmysql34560 psmysql34682 psmysql34693 psmysql34863 psmysql34866 psmysql34870 psmysql34902 psmysql34918 psmysql34923 psmysql34924 psmysql34931 psmysql34932 psmysql35295 psmysql35431 psmysql35614 psmysql35849 psmysql36021 psmysql38574 psmysql38897 psmysql39042 psmysql39134 psmysql39371 psmysql39623 psmysql39671 psmysql39871 psmysql40185 psmysql40224 psmysql40473 psmysql40882 psmysql41002 psmysql41695 psmysql42444 psmysql42965 psmysql43902 psmysql44430

Update 12:07am: Machine has been failed over to new hardware and is up and running again.  Sorry for the downtime.


9:41 am

Mail Downtime on Homie-Sub4

Posted (February 10th, 2011 at 9:41 am PST) by brians

We are currently experiencing problems on one of the filesevers that the mail machines mount to. As a result, you will likely notice a disruption in service ( connection lag / timeout, login errors, delays in message delivery, etc ). Our admins are diligently working towards a solution. Again, this only affects users who are hosted on homie-sub4. You can confirm whether you are affected by clicking the ” Account Status ” link here http://panel.dreamhost.com. Scroll to the bottom and you will see a field labeled ” Your Email Server “. If its set to homiemail-sub4, that would explain any mail problems that you are encountering. We sincerely apologize for any inconvenience this has caused and appreciate your continued patience. Please check back for future updates.

UPDATE 2011-02-10 1:00 PM PST

The problems on the fileserver are not fully resolved as of yet, but the mail machines are looking a little better. We will continue to keep you updated. Also, we cannot provide an exact ETA. But I can assure you that we are doing our best to resolve this matter ASAP. Again, we apologize for any inconvenience this has caused.

UPDATE 2011-02-10 3:11 PM PST

Our administration team have decided that the best course of action would be to move users off of the faulty fileserver. The process of moving nearly 4TB of data can take some time to complete. Therefor you will likely encounter intermittent connection issues during this period. Please continue to bear with us during this time. We would like to thank-you again for hanging in there with us. Please check back for future updates.

UPDATE 2011-02-10 8:28 PM PST

Our administration team has been able to make an adjustment to improve some access times, so all mail on the homie-sub4 cluster should be working now. Users that are actually on this file server may still experience slow mail access times or some time-outs over night, we’re still working on fixing that for good! Hang in there, we’re doing everything we can to get this working correctly!

UPDATE 2011-02-11 9:00 AM PST

The administration team is still working on this matter – we have been offloading the server which has been acting up to take it out of the equation (there are 4 TB of data on there though so this takes time). Due to the fact that offloading is not a quick fix the administration team is still working on other means of addressing the problem and improving performance. They have recommended the following:

-using an IMAP client (not POP3) should improve the performance for you
-for now avoid using webmail (it’s not the most efficient way to access mail under normal circumstances and with the degraded performance right now it’s not the best option)

UPDATE 2011-02-11 8:00 PM PST
We believe we have this issue fixed! We’ve been able to keep a steady stream of traffic to the problem file server, and everything seems to be working just fine! If you’re seeing problems, please let us know and we’ll be happy to check it out for you!


7:43 pm

Shared server ‘ chamaeleon’ having RAID card issues.

Posted (February 5th, 2011 at 7:43 pm PST) by Brian@dreamhost

Oops, looks like chamaeleon is having some difficulty coming back up after a reboot. Our Network Operations Center is addressing the issue and we expect to have the server back up shortly! Thanks for your patience and please contact Support with any questions you have!

Update: chamaeleon is back up and running. If you have any more problems with this server, please contact support.

Update Feb 6th 11:40am (UTC-8): It appears that no action was taken to resolve this issue besides power cycling the machine. Now the RAID card has finally given out and is throwing tons of “arcmsr0: abort device command of scsi id = 0 lun = 0″ errors. One of our techs has been dispatched to replace the raid card. We should have the system back online with a new RAID card in 40 minutes. After that the system may require a filesystem check before resuming normal operation.

Update 12:25p: The RAID card has been replaced and the system is booting. The filesystem is not showing errors so filesystem check will be unnecessary.


10:40 pm

Shared server ‘caelum’ having RAID issues.

Posted (February 4th, 2011 at 10:40 pm PST) by Oscar

This shared server is having some difficulty coming back up after a reboot. Our Network Operations Center is addressing the issue and we expect to have the server back up shortly! Thanks for your patience and please contact Support with any questions you have

Update: Caelum’s RAID array has been recovered and it is booting right now.  All service should be restored in the next few minutes.  Thanks for your patience.

Update 2011-02-07 12:15PM: Caelum has again gone unresponsive and is having RAID issues. Our admins at the lax datacenter are on their way to investigate the problems it is having, and replace the raid card if necessary. When we have more info or an ETA For it to be back up, we will update this post. Our apologies for the repeat issues with this server.

Update 2011-02-07 5:14 PM: I replaced the RAID card about 4 hours ago and have been monitoring it since then. It appears the new RAID card is humming along nicely. If you are still having any issues, contact support.


5:54 pm

Network Maintenance: Friday, February 4th

Posted (February 3rd, 2011 at 5:54 pm PST) by Brian@dreamhost

UPDATE: Maintenance completed

The update is all done. If you are still having issues with your sites being down, please contact our support team. We apologize for the inconvenience this may have caused.

UPDATE: starting now
There will be an outage to reload our core and border routers. This is unavoidable, due to a bug in IOS that is critical for us to patch. We plan to minimize disruption, but would like to stress that you should plan for disruption on service throughout the entire window. The window of downtime should be between 10pm PST and Midnight. We don’t expect to need that whole 2 hour window, but want to be safe in our estimate. Please plan accordingly and thank you for your patience.


10:15 am

Mosrite server going down for RAID card replacement

Posted (February 3rd, 2011 at 10:15 am PST) by danwy

We’re taking down the ‘mosrite’ server for a much needed RAID card replacement. If your site is hosted on this server you may notice a short outage. It should be back up momentarily. We’ll update this status post again once it is complete.

Update – 10:25am (PST): The replacement went smoothly and the server is back up and running again. If you have any further issues on this server, please contact support.


 
© 1996-2011, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
1:55 am

network issues affecting shared and VPS hosting

Posted (February 3rd, 2011 at 1:55 am PST) by patrickm

We are currently experiencing some issues on our network which is causing intermittent slowness and connection issues for shared and VPS hosting.  Our Network Admin has been notified and we are working to resolve this issue as quickly as possible.  We will update this post with more details when we have them!

Update: We found that one of the uplink switches in our LAX facility had degraded itself, which caused the slowness on a particular row of servers. We were able to resolve this issue and all servers on that row are back to normal speed! We apologize for the inconvenience this may have caused and encourage you to contact Support if you have any questions!


6:02 am

Web Panel slowness and access issues

Posted (February 2nd, 2011 at 6:02 am PST) by patrickm

We are currently experiencing an issue with our internal database, which is causing slowness and access issues for the customer Web Panel.  We are working to resolve these issues and will update this post when they have been resolved!

Update: This was caused by an internal bug, which we have resolved!


10:13 am

shared server bishop down for fsck

Posted (February 1st, 2011 at 10:13 am PST) by douglas

Bishop is going down for a filesystem check. It should be back up in a few hours.

Update 17:00 PST: We apologize that the fsck is taking longer than anticipated to complete – unfortunately there’s nothing we’re able to do to force it to complete faster but we are monitoring things to make sure it doesn’t stall. We’ll update this post as soon as it has completed.

Update 19:25 PST: The filesystem check is done and bishop is now back up. If you are having any further issues with your sites being down, please contact our support team. Again, we apologize for any inconvenience this has caused you.


2:38 am

A few shared servers going down for OS upgrade tonight

Posted (February 1st, 2011 at 2:38 am PST) by JeffH

Affected machines are…

burke, rothstein, scalise, schultz, tbilisi and mcgurn.

These will be taken down one at a time, and brought back up once the OS upgrade is complete.  Some or all services may be unavailable for a short time during the process.  We apologize for any inconvenience.

Update – 8:15 am (PST): Unfortunately, the upgrade didn’t go as smooth as we’d hoped. The permissions in the /home directory of the servers were not carried over in the upgrade, so it is resulting in downtime on several of these servers. Our admin team is working on restoring the permissions and hope to have things back to normal ASAP. We apologize for the inconvenience this has caused you and will be thoroughly investigating the cause. We will post another update as soon as we have any additional information.

Update – 10:00 am (PST): Our admin team has just informed us that these servers should be back to normal. Rothstein is still missing some services, but the sites we checked on that are hosted on it were coming up with no issues. We’re still investigating what went wrong and will update this with more information when it is available.

Update – 11:30 am (PST): The issues with these servers should be fully resolved. The final configs have run on rothstein, so all the servers should be functioning correctly again. These were the last handful of ‘etch’ (an older version of linux) servers and they had some weird partition layouts. This is likely what caused the files to not be copied to /home correctly. If you are still seeing errors on your sites hosted on any of these servers, please be sure to clear the cache in your browser to load a fresh copy of your page. Again, we apologize for any inconvenience this has caused you.


10:20 pm

Large scale networking issues affecting a number of customers – RESOLVED

Posted (January 31st, 2011 at 10:20 pm PST) by Jason C

Any DreamHost customers currently running into issues reaching their sites, email or any other services should be aware that a large scale networking problem is messing things up pretty badly at the moment. One of our admins has noted that it looks like a pretty massive denial of service (DoS) attack, but until our network admin has a look at what is going on, that is merely speculation.

As the issue progresses, we’ll update this status post with all of the latest information.

Update 12:31am PST: This situation is now resolved! This turned out to be a DDoS against a site on one of our servers. We have taken steps to stop this attack and everything has returned to normal! Please contact support if you are still experiencing any issues with your hosting. Thanks for your patience!


6:30 am

shared server ‘gambino’ downtime

Posted (January 31st, 2011 at 6:30 am PST) by patrickm

This shared server is having some difficulty coming back up after a reboot.  Our Network Operations Center is addressing the issue and we expect to have the server back up shortly!  Thanks for your patience and please contact Support with any questions you have!

Update: The RAID array was corrupted so we will be failing over from backups. As we have to re-image the machine, we’re using the newer lenny image instead of etch– effectively upgrading the server software.

Update 8:50am: We’re finishing up the final stages of the config process, so we should have backups (and websites) back online “real soon” (tm).

Update 10:00a Web services are starting up now, so you should see you website is either up, or will soon be.

We are running off of the most recent backup of your home directories, so some of your more recent changes may be missing (updates made to a mysql database server will be unaffected).

Update 2/1 12:40pm PSTThe restore is at 40% right now. You may experience longer than usual load times while the restore is running, due to the fact that the data is temporarily being served through the network. Once the restore is done, it’s back to local drives. You may also update your files during this process, as files with newer time stamps won’t be overwritten. Please let support know if you have any questions, and again, apologies for any inconvenience this is causing.

Update 2/2 12:20pm PSTThe restore is at 83%, we’re in the finish line :) Please let support know if you’re having any issues with your sites. One thing to note: the new Gambino machine is a 64bit system, so if you’re using your custom PHP, you may have to recompile it against the new system. Some help in an older status post.

Update 2/3 11:15am PSTThe data restore is almost all done. We’re running only a few more syncs between the old and the new server, and will update this post again shortly, when that finishes. If you’re having any issues with your web service, please don’t hesitate to contact support.


2:23 am

Server, Gemini being failed to new hardware

Posted (January 30th, 2011 at 2:23 am PST) by JeffH

A bit earlier this evening, the shared server, Gemini lost its public network interface, so while the machine is still up, it’s not accessible to the outside world.   We are currently in the process of migrating it over to new hardware, and will post again as soon as it becomes available.  We do apologize for any inconvenience.


8:46 pm

Shared server “monaco” hardware failure

Posted (January 29th, 2011 at 8:46 pm PST) by Brian@dreamhost

We apologize for the downtime caused by this outage. The shared server ‘monaco’ is down due to hardware failure and a tech is moving the machine to new hardware now. We will update this post as soon as we have more information.

UPDATE 1/29/2011 10 p.m. PDT: Monaco is back online and services are being configured, but it will take some time for user data to restore from backup. Our system is restoring data as quickly as possible.

NOTE: You may experience longer load times while the restore is running, as your data is currently being served through the network, not locally.
Also, feel free to update your data during this process, as the data sync will not overwrite files with newer modification time stamps.
If you made changes between the last backup and the crash, unfortunately, you will have to upload the backups again. Sorry again about the inconvenience this is causing. The data restore is 43% completed.

UPDATE 2/1 12:40pm PST: The restore is at 76%, so we estimate another day or so. Thanks for your continued patience, and please let support know if you need help with anything related to this outage.

UPDATE 2/3 11:10am PST: Monaco’s data (from last available backups before the crash) has been fully restored to the new machine. If you’re still having issues with your web service, please contact support with the details, so we can assist you.


10:05 am

Failed sending of mail from scripts

Posted (January 29th, 2011 at 10:05 am PST) by mir

The extended network maintenance last night caused a backlog of email on one of our servers which processes mail send via script (from hosting server users). This did not have any consequences for mail sent through our normal mail system (so mail sent through webmail or mail clients run from customer computers).

Because the vast majority of the mail clogging up the queue was from spam runs our mail expert was forced to completely clear out the queue (because the spam messages have spoofed headers there was no way for us to weed out the legitimate mail quickly enough to prevent the mail clog to be completely out of control). Mail is now getting out but any messages that were in the queue were deleted and will have to be re-sent at this time. We apologize greatly for the inconvenience but it was deemed necessary to avoid allowing a massive quantity of spam to go out from our system (which would have resulted in blacklisting of our servers and more significant mail outages due to blocking of our IPs by recipient mail systems).


8:23 am

Shared server ‘niamey’ hardware failure – RESOLVED

Posted (January 29th, 2011 at 8:23 am PST) by robert

We apologize for the downtime caused by this outage. The shared server ‘niamey’ is down due to hardware failure and a tech is on his way to transfer the drives to another chassis. I will update this post as soon as I have more information.

Update: 10:00am (PST): The server is back. If you are still having trouble with this machine, please be sure to contact our friendly support staff!


 
© 1996-2011, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
9:30 pm

Network Maintenance: Friday, January 28th – 22:00 through Saturday, January 29th 02:00 PST

Posted (January 28th, 2011 at 9:30 pm PST) by robertn

There will be a 15 minute outage to reload our core and border routers. This is unavoidable, due to a bug in IOS that is critical for us to patch.

Plan on a 15 minute outage at our Downtown facility at 22:00 PST.
Plan on a 15 minute outage at our LAX facility at 00:00 PST (Saturday, January 28th)

Also at our LAX facility we will reload several of our distribution and access switches. This could also result in a 10 minute disruption in service, but we are endeavoring to coordinate that effort with the core reload to minimize downtime.

Please plan accordingly.
Thank you for your patience

Update 11:55PM (PST): We’ve gone longer than we’ve been expecting and our Network staff is currently continuing to update our infrastructure. We will update you with further information asap and we’ll have you back online as soon as possible.

Update: 12:40AM (PST) We have partial connectivity and that we are working with Cisco’s support (TAC) to resolve this issue permanently. We are going to be extending the window an additional hour to allow them to debug and solve this issue.

Update: 02:07 (PST) We’ve restored connectivity and we’re currently in the process of cleaning our network that has been affected. This may be seen on your website as MySQL downtime and our admin staff is currently fixing that up.

Update: 5:50am (PST) While overall network connectivity has been restored, there’s a single rack of servers that is still experiencing an outage following the network maintenance.  We currently have our network engineer and admins working on restoring service, but there’s a variety of machine types in this rack, from MySQLs to VPS servers.  We appreciate your patience and sincerely apologize for the downtime you’ve experienced.

Update: 7:50am (PST) Sorry for the delay in updating. All of the servers are accessible now except for one, the server niamey is still down at this time and we are working as quickly as possible to get it back up.

Update: 8:15am (PST) I just made a separate post regarding niamey at http://www.dreamhoststatus.com/2011/01/29/shared-server-niamey-hardware-failure/  as its downtime is unrelated to the networking issue.


2:42 pm

Gambino raid card issues – RESOLVED

Posted (January 26th, 2011 at 2:42 pm PST) by danwy

The Gambino server is currently having some issues due to its RAID card dying. We’re doing what we can to keep things running until we can put a new RAID card in. The server is hobbling along now and there will be an outage when we have to take it down to replace the malfunctioning hardware. We’re currently expecting it to take 30 minutes to be back to normal. We will update this post as soon as we have any additional information, or when the problem is resolved. We’re sorry for any inconvenience this causes you.

UPDATE 28 Jan: Sorry for the delay in updating. Shared server gambino is now up and running. If you still experience issues on shared server gambino, please contact Technical Support staff through the normal channels who will cheerfully help you resolve your issues.


12:08 am

Shared server ‘warsaw’ hardware failure, failover….(it’s down)

Posted (January 25th, 2011 at 12:08 am PST) by JeffH

Nothing lasts forever.  In this case, the point is illustrated by the failure of the hardware we’ve come to know and love as “warsaw”.  We’ve dispatched a tech to transfer the drives to another chassis however, and it should be back up in a bit.  Thankfully, it doesn’t look like we’ll have to restore from backups, so the process should be relatively swift.  We’ll update this post as the situation matures.

Update: My apologies!  I forgot to update this post last night, but this machine was successfully failed over to new hardware within two hours of this status post’s original timestamp.  The drives from the original machine were fine, so there was no need to restore from backups.  Everything should be back up and running now, but if you’re still having problems, please let us know via an official support channel (web panel, chat, NOT comments here).    Thanks for your patience!


10:59 pm

Vserver host ‘homie-vserver165′ down for hardware maintenance

Posted (January 23rd, 2011 at 10:59 pm PST) by Ken

We regret the inconvenience that this is causing. A technician has been dispatched to resolve the problem. This post will be updated as soon as we have more information available.

Affected VPS guests:
ps23353 ps23467 ps23640 ps23674 ps23712 ps23744 ps23783 ps23833 ps23914 ps23957 ps24126 ps24168 ps24842 ps24858 ps24929 ps25011 ps25355 ps25439 ps25528 ps25541 ps25719 ps25798 ps25816 ps25980 ps26298 ps27366 ps27745 ps28430 ps28437 ps28443 ps28453 ps28454 ps28538 ps28665 ps28678 ps28681 ps29032 ps29224 ps29237 ps29634 ps29674 ps29772 ps29798 ps29801 ps29904 ps30034 ps30102 ps30250 ps30286 ps30555 ps30649 ps31151 ps31204 ps33408 ps33451 ps33646 ps35337 ps35557 ps35796 ps35837 ps35941 ps38168 ps38178 ps39342 ps41653 ps41680 ps41701 ps42328


7:51 am

Moving MySQL servers in the frisky cluster

Posted (January 23rd, 2011 at 7:51 am PST) by robert

We are moving MySQL servers in the frisky cluster to a new rack in the next 15 minutes. Downtime for each MySQLs should only be 5-10 minutes and a list of MySQL servers affected is below:

dangermouse
nicodemus
algernon
stuartlittle
minnie
mightymouse
splinter
mickey
pukka
gus
itchy

Sorry for the inconvenience. I will update this post once the move is finished.

Update 9:01AM: The MySQL servers listed above were moved and are up at this time. Please contact Support if you are still experiencing any issues!


9:00 pm

Network Maintenance: Saturday, January 22nd — 21:00 -23:59 PST

Posted (January 22nd, 2011 at 9:00 pm PST) by robertn

UPDATE: Completed successfully

UPDATE: Moving this to Saturday, January 22nd — 21:00 through 23:59

We will be adding some hardware to our existing core and border infrastructure. This should not cause any interruption, as all this gear is standby gear.
HOWEVER, We will be upgrading the IOS in several of the rows which WILL cause a minor interruption in service. Please contact support if you notice an outage longer than 20 minutes.

The following servers will experience brief connectivity issues while the switches reload:

K-row
alvin, sloop, simon, theodore, mickey, jerry, ratbert, pukka, mortimer, itchy, gus, vserver-img-test, samurai, dangermouse, splinter, minnie, mightymouse, stuartlittle, paddington, nicodemus, algernon

Z-row
homie-vsql44, homie-vsql45, nemo, megalodon, mako, wobbegong, silvertip, porbeagle, hammerhead, nursehound, homie-vsql47, swellshark, wodnika, knifetooth, chimaera, homie-vsql46, homie-vsql43

X-row
peppy, homie-vserver174, homie-vserver172, homie-vserver173, chisinau, skopje, monaco, kualalumpur, bamako, valletta, majuro, portlouis, mexicocity, palikir, lilongwe, ulaanbaatar, podgorica, rabat, maputo, rous, falco, slippy, ecco, starfox, dedede, andross, wario, daisy, nelly, hippo, hambone, vamika, flint, spacey-vsql05

C-row
homie-vserver175, homie-vserver176, pretoria, mogadishu, ljubljana, yangon, homie-vserver162, homie-vserver163, homie-vserver164, homie-vserver165, homie-vserver166, homie-vserver167, homie-vserver168, homie-vserver169, homie-vserver158, homie-vserver151, homie-vserver150, homie-vserver149, homie-vserver148, homie-vserver147, homie-vserver145, homie-vserver159, homie-vserver160, homie-vserver161, castries, apia, kigali, moscow, bucharest, doha, lisbon, warsaw, manila, panamacity, homie-vserver157, homie-vserver156, lima, homie-vserver155, homie-vserver154, homie-vserver153, asuncion, kingstown, wellington, managua, melekeok, islamabad, muscat, oslo, abuja, niamey, victoria, rangoon, nouakchott, riyadh, belgrade, windhoek, kathmandu, amsterdam

L-row
names were omitted to protect the innocent

J-row
frisky-vsql02, frisky-vsql01, swarthy-vsql01, looney-vsql01

Thank you for your patience.


6:33 am

Shared server ‘abudhabi’ downtime for hardware maintenance – RESOLVED

Posted (January 22nd, 2011 at 6:33 am PST) by robert

We apologize for the downtime caused by this outage. The shared server ‘abudhabi’ is down for RAID card replacement at this time.  We expect the service to be restored quickly and will update this post when that happens!

Update 6:42am PST: This maintenance has been completed and the server is now back up and running.  Please contact Support if you are still experiencing any issues!


2:35 pm

Networking problems causing downtime for Mysql RESOLVED

Posted (January 21st, 2011 at 2:35 pm PST) by Oscar

We’re currently experiencing some networking issues with one of our racks in our LAX datacenter. We are working to resolve the issue and will update this post as soon as more information is available.

Update: 2:55pm PST: There was a power strip failure which took down the switch. We are redistributing the power load now. Affected machines are:

homie-vsql61 homie-vsql62 homie-vsql63 homie-vsql64 homie-vsql65 homie-vsql66 jabberjaw homie-vsql70 sawshark lanternshark shyshark bonnethead  homie-vsql67 homie-vsql68 homie-vsql69 snap misha

Update 4:22pm PST: This issue is completely resolved.  We replaced the bad power strip with a new one and all of the machines are back up and running. If you have any further issues with your sites or databases being down, please contact our support team.


5:01 am

Vserver host ‘homie-vserver166′ down for hardware maintenance – RESOLVED

Posted (January 21st, 2011 at 5:01 am PST) by patrickm

We apologize for the downtime caused by this outage.  We are replacing the RAID card on this host server and expect it to be operational soon!  Affected VPS guests:

ps23438 ps23468 ps23561 ps23597 ps23675 ps23713 ps23834 ps24127 ps24259 ps24337 ps24380 ps24557 ps24821 ps24855 ps24863 ps24902 ps24960 ps25031 ps25103 ps25247 ps25373 ps25685 ps25740 ps26641 ps26645 ps26647 ps27929 ps27966 ps28153 ps28787 ps28809 ps28835 ps29210 ps29220 ps29246 ps29561 ps30192 ps30230 ps30232 ps30237 ps30691 ps30726 ps30907 ps33473 ps33511 ps33516 ps33550 ps34083 ps34098 ps34447 ps35551 ps37059 ps41452 ps41606 ps41610

Update 5:15am PST: This has been completed and the host server is back up and running.  If your VPS is not functioning, or you notice any other issues, please contact Support!


4:04 am

Network Outage

Posted (January 21st, 2011 at 4:04 am PST) by Sandon

About 20 minutes ago (3:40 AM PST) we had a network outage was making most of our network inaccessible. Things appear to be coming back up now and our network engineer is investigating the cause.

Update 06:30 AM PST It looks like we were not notified of a maintenance that was being done on one of the multiplexer’s that one of our transit providers in our Down Town Datacenter goes through which caused the trouble. Excess Traffic was automatically routed through our second DC (LAX) but the transition was not as smooth as it should have been and caused network connectivity issues for about 20 minutes.

As of about 5:20 AM we fixed the connection that went down and traffic transitioned back over to the link without any issues/outages. We have been keeping an eye on the network for the last hour and there have been no problems with connectivity since.


 
© 1996-2011, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
3:12 am

Shared server ‘abuja’ downtime for hardware maintenance – RESOLVED

Posted (January 21st, 2011 at 3:12 am PST) by patrickm

We apologize for the late notice on this and the inconvenience this downtime has caused.  The shared server ‘abuja’ is down for RAID card replacement at this time.  We expect the service to be restored quickly and will update this post when that happens!  Thanks for your patience and please contact Support if you have any questions.

Update 5:18am PST: This maintenance has been completed and the server is now back up and running.  Please contact Support if you are still experiencing any issues!


1:56 pm

Dynamo requiring fsck – RESOLVED

Posted (January 20th, 2011 at 1:56 pm PST) by mir

The admin team had to reboot this server and it is currently running fsck (http://en.wikipedia.org/wiki/Fsck). The admin team is monitoring it and will update this post when the check has completed and the server is fully functional again (the ETA is 3-4 hours).

Update (14:00 PST) It looks like the disk is bad so we’re actually going to fail it over onto new hardware which should only take around 30 minutes. We’ll update this post again as soon as that is completed.

Update (16:00 PST) We apologize that the initial estimate was overly optimistic but are almost done with the task of moving dynamo to new hardware. Currently we’re in the final stages of getting the users configured (about 15 minutes) after which we should be able start restoring services (hopefully completed in the next 45-60 minutes).

Update (00:52 PST) This process of restoring data to the new server is still underway.  We are working to balance the data transfer with the normal operation of the server, so we apologize for any downtime still being experienced.  We will update this post again when we have additional information, or when this process is completed.  Please feel free to contact Support if you have any specific questions, or require assistance!

Update (09:20 PST) We hit a minor bump in the restore process where we realized some users content was not properly being shown. The restore itself was not interrupted but some files were just not being served correctly. We have corrected that problem so if your sites were not working prior to 8:00, they should be now.

Update 1/23 4:50pm PSTThanks for your continued patience while the restoration is ongoing. We’re about half way through, but the sites should be working fine, running off the last backup before the machine crashed. If you updated anything between the last backups and the crash (ie files that look old or missing now), please upload them again. You may also update files during this process, as the restoring script will not overwrite data with newer time stamps. We’ll update this post again if there is any more progress. Sorry again about all the inconvenience this is causing.

Just a quick note: because of all the data syncing processes, and because your data is currently being served through a network file system instead of being local (which is what it was and will be once the process is done), your load times will be longer. Apologies for the inconvenience!

Update 1/25 11:25am PSTJust a quick update to let you know that the restore is at now 80%. We’ll update this post again as soon as it’s complete. Thanks for your continued patience!

Update 1/26 19:15am PSTThe restore to the new machine is now complete. The load times should go back to normal, and everything else should be functional as well. Please contact support if you need further assistance with your domains on this machine.


10:39 am

Mail problems on homiemail sub5 again

Posted (January 20th, 2011 at 10:39 am PST) by danwy

We are experiencing some issues with our homiemail-sub5 mail cluster again that is causing mail outages. The last time this happened it was due to a switch problem, which we suspect is the issue again. Our mail admin is currently on his way down to the datacenter to get resolve this problem. You can tell if you’re on the sub5 mail cluster by going to the ‘manage domains’ section of your panel and clicking the ‘DNS’ link for any of your domains. If your MX records show up with the following, then you are likely affected by this issue:

mx1.sub5.homie.mail.dreamhost.com.
mx2.sub5.homie.mail.dreamhost.com.

Please note, if you’re using Junk Filter on your domain, the MX records will show the filtering servers, but your mail may still be using the homiemail-sub5 servers for delivery. In this case, if mail.yourdomain.com resolves to 208.113.200.129, you’re affected. You can check this under Domains/Manage Domains/DNS link.

We will update this status post as soon as we have more information.

Update – 12:30pm PST: This issue didn’t end up causing as many issues as we thought it was. We thoroughly checked the switch and didn’t find any problems. We now suspect that it was just a surge in traffic. It may have caused mail to be a little slow, but it didn’t cause any large scale outages like we expected. Our admin team is going to continue to investigate the issue and make adjustments where they see fit to make sure things remain stable. If you have any further problems with mail, please contact our support team to investigate it further.


9:15 am

kigali maintenance – RESOLVED

Posted (January 19th, 2011 at 9:15 am PST) by douglas

kigali’s RAID card is being temperamental, as a precaution we’re dispatching one of our techs to replace the RAID card.

Update: The RAID card has been replaced and is behaving properly.


4:14 am

Shared servers ‘windhoek’ and ‘bootes’ downtime due to hardware issues – RESOLVED

Posted (January 17th, 2011 at 4:14 am PST) by brettg

Both windoek and bootes servers are experiencing hardware failures at the moment. Our technicians are currently investigating why they are failing and we’ll bring your more status updates as we go.

Update 8:00 am: Sorry for the delay. Both machines should be back up now. Bootes was stuck in a reboot loop and Windhoek’s motherboard died. If you’re noticing further problems, please contact our support staff via the Web Panel.

Update 1/18/11 5:19am Pacific: Shared server ‘bootes’ is down again for a RAID card replacement. Service will be restored shortly!

UPDATE 01/18/0531 Pacific: Shared server bootes is marching on again. If you still have issues, please contact Technical Support through the normal channels


4:05 pm

Shared server ‘astana’ getting a new raid card – RESOLVED

Posted (January 15th, 2011 at 4:05 pm PST) by mir

Our admin team noticed a problem with the file system again on this machine so they are replacing the hardware (raid card) in order to fix it for good. Downtime should be minimal (around 15 minutes).

Update 16:32: It looks like we need to get this server onto entirely new hardware so it is being failed over now which should fix it up once and for all. This should be completed within 30-45 minutes (we’ll update again when complete).

Update 19:41: We certainly apologize for the delay on this, due to complications, we are still working to get services restored.  We will be sure to update this post as soon as services are restored.  Please contact our support staff if you have any questions!

Update 22:30: Services on Astana are being restored at this time, and data is copying over from backups.  The data restoration itself may take a couple days, but during that time the server will remain up and running as the backups are mounted locally.  Sites should be available in the next one or two hours…our graveyard staff is taking over and will be monitoring the situation to be sure everything goes smoothly.

Update Jan 18th: The data restore has finished and all services are running normally.  If you are still having trouble, please be sure to contact our friendly support staff!


1:43 pm

Shared server ‘astana’ maintenance

Posted (January 13th, 2011 at 1:43 pm PST) by Oscar

There is an issue with a corrupt journal on /home/filesystem. We’ve removed the journal are currently running a filesystem check. Once fsck is done, we’ll rebuild the journal and get ‘astana’ back up. We’ll update this post as soon as we have more information.

Update 16:19: Astana is back amongst the living. If you’re still having problems with this machine, please contact your nearest ever-ready DreamHost support person.


11:29 am

Issues with homiemail-sub5 mail servers

Posted (January 12th, 2011 at 11:29 am PST) by danwy

We are currently experiencing some issues with our homiemail-sub5 mail cluster that is causing mail outages. We’re not sure what the cause is yet, but our mail administrator is currently investigating it. You can tell if you’re on the sub5 mail cluster by going to the ‘manage domains’ section of your panel and clicking the ‘DNS’ link for any of your domains. If your MX records show up with the following, then you are likely affected by this issue:

mx1.sub5.homie.mail.dreamhost.com.
mx2.sub5.homie.mail.dreamhost.com.

Please note, if you’re using Junk Filter on your domain, the MX records will show the filtering servers, but your mail may still be using the homiemail-sub5 servers for delivery. In this case, if mail.yourdomain.com resolves to 208.113.200.129, you’re affected. You can check this under Domains/Manage Domains/DNS link.

We will update this status post as soon as we have more information.

Update – 12:20pm (PST): This issue has been resolved and mail on this server is functioning correctly again. Somehow the switch for the cluster lost its config file and caused a whole variety of problems. We restored the config file file and things are much better now. We’ve tested dozens of affected email addresses and verified that they’re now working. We apologize for any inconvenience this caused you. If you continue to have mail issues, please contact support, as it is not related to this issue.


10:17 am

Personal backup server overage charges – announcement

Posted (January 12th, 2011 at 10:17 am PST) by andrea

Charges for overage on the backup server were initially not enabled, as the backup server was unstable for some time after it was set up. However, the backup server has now been working properly for some time now, so we have made the decision to start charging for usage in excess of the limit, starting with your February billing cycle.

Those who are currently above the free limit or are getting near it should soon receive a notice with details pertaining to their own account status, including the estimated charges based on current usage. As a reminder, your personal backup service includes 50GB of data for free, and is charged $0.10/GB/month for any usage which exceeds the limit. When determining if you’re over your 50GB limit, we take your usage for that billing cycle and average it. The average depends on the sampling taken of the data usage during the cycle. We’ll also send out warnings to advise if you should be expecting overage charges in that billing cycle.

Not to worry — we will not be billing you retroactively for any overage that occurred up until now. We will only be assessing charges for overages which occur after the start of the February billing cycle.

You can check your current usage in the Panel under Users: Backups User. Note that the usage as displayed in the Panel may take some time (up to a week or so) to update with any changes you make — if you’ve just added or deleted a large amount of data and need to check on your exact usage, let us know.

The description and conditions of the service can be found on our wiki under http://wiki.dreamhost.com/Personal_Backup. Please contact support if you have any questions.


12:25 pm

Shared webserver ” Cancer ” is down

Posted (January 11th, 2011 at 12:25 pm PST) by brians

This is affecting all access ( http, ssh/ftp, etc. ) to the machine. Our administration team is in the process of replacing the hardware on the machine. Unfortunately, we do not have an ETA at the moment. I can assure you that we are diligently working towards getting the machine back online ASAP. We sincerely apologize for any inconvenience this has caused and appreciate your patience. Please check back for future updates.

UPDATE 2011-01-11 4:51PM PST:

All services are back online. However. it appears that we encountered a RAID degrade. Have no fear. Your data is in tact. We have mounted the machine to a backup drive. Data is also being copied to home directories as I type. The process of copying the data will likely take a couple of days. You might encounter intermittent stability issues during this time. Again, we sincerely apologize for any inconvenience this has caused. Please check back for future updates.

Update 2011-1-12 11:35AM

Through much effort , we have fortunately been able to recover the raid array, and are in the process of restoring the previous hardware.  We are also in the process of syncing up the data between two machines, to ensure that any files that were uploaded within the past 18 hours are not lost. Sorry again for the inconvenience this has caused all of you!


 
© 1996-2011, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
7:07 pm

Slowness on DreamHost web panel

Posted (January 9th, 2011 at 7:07 pm PST) by Jason C

Some customers may find that panel.dreamhost.com is either slow or inaccessible when they try to visit it.  Our admin team is currently looking into the matter and is doing everything they can to make things snappy again.  As soon as everything has leveled out, we’ll have an update to this post.

Update: Our admins tell me that this issue was caused by a stuck query on our central database server.  As of 7:22 PM (PST), the panel should be fully accessible again.  If you have any further issues, please do not hesitate to contact our support team!


8:27 am

Shared server “auriga” down for hardware maintenance — Resolved

Posted (January 9th, 2011 at 8:27 am PST) by Justin@DH

The shared server auriga is showing raid card errors that are re-occurring even after reboots, so our admin team is on the job to replace its raid card. When we have further info such as when it will be fixed or when it is fixed, we will update this post. Our apologies for the downtime occurring for those of you on this server.

Update 9:04AM PST – The raid card has been replaced and the server is working normally again. We will be keeping an eye on it for further issues. If you still have any downtime or issues and you are on this server, please contact support.


9:00 pm

Network Maintenance: January 8th, 2011 21:00 – 23:30

Posted (January 8th, 2011 at 9:00 pm PST) by robertn

UPDATE:  This maintenance has been completed.

Thank you for your patience.

We will be doing some additional maintenance on our Border routers to improve our connectivity tonight.

We also have to reload two of our routers, which will cause a brief outage.  Certain email accounts could be down for 5-10.  These servers could also experience a brief outage:

P row:

buster, cabbit, gordo, wolpertinger, gizmo, furball, gongon, meemee, aiai, baby, chimchim, clyde, mojo, kingkong, shamu, droop, searobin, gurnard, albacore, sturgeon, barracudina, homie-vsql71, barbel, killifish, wrasse, ape, rafiki, randy-vsql03, randy-vsql01, randy-vsql02, koko, littleredmonkey, diddykong

Q row:

conklin, ltd, squier, univox, zon, ampeg, mesaboogie, bogner, diezel, engl, hiwatt, laney, marshall, alembic, randall, sunn, carvin, luanda, caparison, bcrich, kalamazoo, cort, danelectro, dobro, dean, epiphone, esp, ibanez, ernieball, fernandes, godin, gretsch, guild, hagstrom, fender, jackson, charvel, johnson, firstact, samick, jayturser, michaelkelly, memphis, austin, alvarez, dillion, jbplayer, warmouth, ravenwest, hohner, series10, gtx, yamaha, valleyarts, robelli, hamer, harmony, hofner, kramer, hondo, line6, cfmartin, mosrite, ovation, paulreedsmith, peavey, rickenbacker, schecter, steinberger, tobias, vox, warwick, washburn, traceelliot, voodoo, yorkville, framus, kay, ashdown, budda, crate, mudskipper, tech21, soldano, hartke, krank, matchless, sculpin


10:00 pm

Network Maintenance: January 7th 22:00 – January 8th 04:00 PST

Posted (January 7th, 2011 at 10:00 pm PST) by robertn

UPDATE:  The IOS upgrade went as planned.  No site was down for more than 10 minutes

The traffic engineering was impacted by one of our providers going off-line for approximately 2 hours.  This caused intermittent (and not so intermittent by some of your comments) routing issues which had some sites unreachable by some segments of the internet.  This work has been completed.  If you are experiencing an issue please contact support immediately.

Thank you for your patience.

We will be doing some traffic engineering on our border routers and upgrading the IOS version on several of our distribution routers during this maintenance window.

The traffic engineering will insure that one link does not get saturated during failure of another provider and that traffic will get split evenly over the three remaining providers.

The IOS upgrade will involve between 5 and 10 minutes of downtime while the hardware reboots. This will involve the Panel and some mail clusters. A detailed list of servers will be posted below:

B row:

galaxy, clippers, rams, sparks, blazers, homie-vserver57, homie-vserver58, homie-vserver59, dodgers, homie-vserver60, butterfish, lakers, raiders, kings, bonito, tetra, mackerel, anchovy, puffer, narwahl, dory, vpsh2lenny, gill, homie-vsql31, homie-vsql32, homie-vsql33, coelacanth, homie-vsql35, homie-vsql36, salmon, halibut, nanook, marbles, janky-vsql01, huey, dewey, woodstock, tweety, daffy, donald, magic, mariners, warriors, mets, redwings, basking, homie-vserver61, homie-vserver62, angels, homie-vserver64, homie-vsql19, homie-vsql20, eltigre, foghorn, homie-vsql39, padres, niners, seahawks, tigers, steelers, lions, homie-vserver66, homie-vserver65, homie-vserver67, homie-vserver68, homie-vserver69, knicks, homie-vserver70, bubsy, homie-vsql22, homie-vsql23, homie-vsql24, suns, diamondbacks, broncos, braves, patriots, hornets, rockies, slowpoke, leghorn, bruins, inki, homie-vsql21, titans, magoo, grouper, jetsons, gumby, topcat, solomon, spaceghost, homie-vsql38, dumbo, chapulin, zazu, pumbaa, dopey, gurgle, popeye, bullwinkle

K row:

violence, isis, gauntlet, homie-vserver143, homie-vserver144, homie-vserver142, vilnius, vaduz, tripoli, monrovia, maseru, luxembourg, seahorse, freetown, singapore, homie-vserver140, wahbackup, bratislava, pyongyang, homie-vserver141, homie-vserver134, homie-vserver135, homie-vserver136, homie-vserver137, homie-vserver138, seoul, homie-vserver130, pristina, kuwaitcity, bishkek, riga, vientiane, beirut, homie-vserver131, homie-vserver132, homie-vserver133, kampala, kyiv, abudhabi, london, homie-vserver181, homie-vserver182, homie-vserver183, homie-vserver179, homie-vserver180, vip-vserver01, homie-vsql37, ankara, ashgabat, baghdad, dublin, jerusalem, tokyo, amman, nairobi, rome, kingston, astana, homie-vserver128, homie-vserver127, homie-vserver129, tarawa-atoll, tarpon, guppy, jakarta, tehran, conakry, athens, bissau, georgetown, accra, port-au-prince, tbilisi, tegucigalpa, banjul, budapest, reykjavik, newdelhi, homie-vserver124, homie-vserver125, homie-vserver126, saintgeorges, guatemalacity, berlin

V row:

peabody, huckleberry, topogigo, roadrunner, lady, santaslittlehelper, madcat, lily, deputydawg, homie-vsql25, tib, sylvester, flash, homie-vsql15, homie-vsql17, homie-vsql16, fatalbert, shaggy, elmerfudd, yogi, spongebob, sardine, tuna, cod, swordfish, homie-vsql11, homie-vsql42, homie-vserver189, homie-vsql12, homie-vsql14, hanjin-shelf4, hanjin, felix, homie-vsql10, homie-vsql18, yellowtail, homie-vserver04, homie-vserver05, escolar, homie-vserver06, flounder, homie-vserver02, halosaur, homie-vsql58, homie-vsql59, terrance, phillip, towelie

X row:

homie-vserver104, homie-vserver105, homie-vserver146, trout, perch, moroni, brazzaville, kinshasa, sanjose, yamoussoukro, zagreb, havana, helsinki, paris, homie-vserver101, libreville, homie-vserver178, homie-vserver102, homie-vserver103, djibouti, roseau, santodomingo, dili, quito, cairo, sansalvador, malabo, asmara, tallinn, addisababa, suva, homie-vserver120, wahoo, nicosia, prague, homie-vserver119, homie-vserver110, homie-vserver111, homie-vserver112, homie-vserver113, homie-vserver114, homie-vserver116, homie-vserver117, homie-vserver118, copenhagen, homie-vserver123, homie-vserver121, homie-vserver107, homie-vserver122, homie-vserver106, minnow, homie-vserver108, homie-vserver109, tammany, elmyra, tachibana, sarabi, drooper, jingoro, ragland, streaky, skimbleshanks, godfather

Z row:

postal-vsql01, balto, imu, gromit, alc-maildb2, grimm, janky-vsql03, janky-vsql02, alc-maildb1, nipper, dynomutt, einstein, slinky, precious, checkers, zero, schnoodle, oneclick1, slaughter, oneclick2, homiemail-a64, homiemail-a70, homiemail-a71, homiemail-a72, homiemail-a73, homiemail-a74, homiemail-a75, homiemail-a76, homiemail-a7, homiemail-a60, homiemail-a61, homiemail-a5, homiemail-a62, homiemail-a63, homiemail-a66, homiemail-a35, homiemail-a67, homiemail-a29, homiemail-a28, homiemail-a30, homiemail-a31, homiemail-a32, homiemail-a27, homiemail-a6, homiemail-a65, homiemail-a34, homiemail-a33, homiemail-a26, homiemail-mx19


10:51 am

SquirrelMail is down

Posted (January 7th, 2011 at 10:51 am PST) by brians

All ” web-based ” e-mail access ( WebMail, Roundcube, etc ) is down at the moment. Our admins are currently investigating the matter. In the meantime, you can continue to use regular mail clients ( Outlook, Thunderbird, etc ) for mail access. Sorry for any inconvenience this has caused. Your patience is greatly appreciated. Please check back for future updates.

UPDATE 2011-01-07 3:09PM PST:

The issues with WebMail was resolved shortly after my initial post. There were inflated loads on the machines. That cause the stability issues that you encountered. The problem corrected itself. Meaning, the loads went down without any work performed on our behalf. Sorry if this update appears to be anti climactic. However, we will continue to actively monitor the stability of the machines.


1:32 am

Shared server ‘hohner’ down for hardware maintainence – RESOLVED

Posted (January 7th, 2011 at 1:32 am PST) by patrickm

The RAID card in the shared server ‘hohner’ is currently being replaced, which is causing downtime for customers on this server.  We do not expect this process to take very long and will update this post when it has been completed!  We apologize for this inconvenience!

Update 1:52am Pacific: This process is now complete and the server is up and running again.  If you continue to experience any issues, please feel free to contact Support!


10:50 pm

Homie-vserver15 going through drive checks **RESOLVED**

Posted (January 6th, 2011 at 10:50 pm PST) by Ken

May take a couple of hours. We thank you for your patience while we work through this issue. Affected PS’s are:

ps14207, ps14219, ps14221, ps14307, ps14314, ps14476, ps14782, ps14800, ps14982, ps14989, ps15056, ps15100, ps15964, ps16706, ps16707, ps17094, ps20150, ps28889, ps28911, ps28956, ps30149, ps30645, ps35136, ps35460, ps35594, ps35806, ps36243, ps37003, ps37051, ps37264, ps37272, ps38002, ps38079, ps38126, ps5161 ,ps5162, ps5163, ps5165, ps5166, ps5173, ps5177, ps5189, ps5193, ps5196, ps5200, ps5530, ps5541, ps5578, ps5579, ps5766, ps5771, ps6161, ps6307, ps6309, ps6310, ps6311, ps7506, ps7673, ps8129, ps8208

UPDATE Jan 06 2330PST: It looks like we got lucky and the filesystem check has finished. If you continue to experience issues, please contact Technical Support via the normal channels.


10:18 am

Shared web machine ” gacrux ” is down — Resolved

Posted (January 3rd, 2011 at 10:18 am PST) by brians

The machine is experiencing hardware problems at the moment. Our admins are diligently working towards getting the machine back online. We sincerely apologize for any inconvenience the downtime has caused. Please check back for future updates.

Update 11:25am:The hardware has been replaced by a machine that doesn’t fry ram modules.


1:50 pm

WordPress 3.0.4 auto upgrades on hold until further notice

Posted (December 30th, 2010 at 1:50 pm PST) by andrea

If you have previously elected to get your WordPress one-click install to auto upgrade when we release a new version, unfortunately, that didn’t happen yet with this version. If you wish your install to be at the latest version asap, please visit your panel to initiate the upgrade manually, until further notice.

We made 3.0.4 available yesterday, but we’re holding off running the auto upgrades until we can ensure that it won’t back up our other updates running on the system. The solution is being coded as I’m typing this, and we’re hoping to start the auto scheduling later today, but for now, your best bet for a fast upgrade is by going to Goodies/One-Click Installs in your panel and clicking the upgrade button yourself.

Please check back for updates, and apologies for the inconvenience.


1:25 pm

shared server ‘ovation’ downtime due to hardware issues

Posted (December 30th, 2010 at 1:25 pm PST) by douglas

We sincerely apologize for the downtime being experienced on the shared server ‘ovation’.  The server did not return from a reboot and we are now in the process of moving this server over to new hardware.  We estimate that this will take about 30 minutes.

Update 1:45pm: This has been completed.  The server’s hard drives were successfully migrated to a new machine and the server is back up and running.  If you are experiencing any issues, please contact Support and we will be happy to investigate!  Thanks very much for your patience!

Update 4:19pm: Actually, seems there might still be some lingering issues. we’re working on that, sorry about the continued issues!

Update 6:15pm: Okay, NOW we have ovation sorted out. Sorry about that!! Won’t happen again. If you’re still having problems with this machine, please feel free to contact your helpful DreamHost support technician.


 
© 1996-2010, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
4:51 am

vserver host ‘homie-vserver156′ downtime for RAID card replacement – RESOLVED

Posted (December 30th, 2010 at 4:51 am PST) by patrickm

We sincerely apologize for the continued issues with this vserver host, which has caused downtime for the VPS guests over the last week.  Due to an issue with the RAID card this morning, we have taken down the server and we are in the process of replacing the RAID card hardware.  This process shouldn’t take very long and we will update this post with the details once they are available.

Affected VPS guest servers are:

ps23332 ps23341 ps23361 ps23385 ps23428 ps23459 ps23550 ps23683 ps23861 ps23996 ps24062 ps24439 ps24493

Update 5:15 am Pacific: The RAID card replacement has been completed and this host server is now back up and running.  If you are having any issues, please contact Support for assistance!


12:24 am

Shared server ‘gaborone’ maintenance – RESOLVED

Posted (December 30th, 2010 at 12:24 am PST) by Ken

There is an issue with the raid drive on shared server gaborone. Please bear with us while we check it out.

We will update this post as soon as more information becomes available.

Shared server gaborone is currently running a filesystem check (fsck) and may take several hours to complete. Again, thank you for your patience while we work to resolve this issue.

Update 4:45am Pacific: We have completed this file-system check on ‘gaborone’!  The server is now back up and running, so please contact support if you are experiencing any issues with your hosting!


4:58 pm

WordPress hack cropping up on some customer sites.

Posted (December 29th, 2010 at 4:58 pm PST) by Jason C

A noticeable amount of customers who have not yet upgraded their copies of WordPress to the most recent version (3.0.4) have been reporting issues with logging into their dashboard. Upon further review, most of these customers have had code inserted into a large number of WordPress files.

The 3.0.4 update was made available for WordPress earlier today — and is available for upgrade via our one-click installer — but based on the timestamps on a number of infected files, the hack has been ramping up over the past week. This means users with any version of WordPress before 3.0.4 should upgrade their sites immediately using either our installer or by logging into wp-admin and going to “Dashboard > Updates”.

If you find that your site is hacked, an update to WordPress 3.0.4 won’t actually do much. This is because the inserted code ends up in both plugins & themes and its payload is stashed away in a file that does not get overwritten in an upgrade.  So if you’re stuck with a hacked site, please don’t hesitate to contact our support team for advice on clearing things up.


11:33 am

Gabarone going down for file system check

Posted (December 29th, 2010 at 11:33 am PST) by Leon

D’oh! Gabarone has an issue where it’s not allowing anyone to write to disk, so we’re taking it offline for a few hours to run a file systems check. Sorry about that! We’ll keep you updated on how it’s doing when we know.

Update: file system check has entered the second phase, so we’re halfway done now! We’ll continue to update.

Update: file system check is complete and the machine is back online. If you have any questions or concerns, please contact your friendly DreamHost support cyborg.


9:02 am

basking mysql server going down for maintenance

Posted (December 28th, 2010 at 9:02 am PST) by douglas

basking is going down for maintenance to add some extra hardware to it.

Update: Maintenance is complete.


6:40 am

homie-vserver 01 fsck

Posted (December 23rd, 2010 at 6:40 am PST) by Sandon

Homie-vserver01 is currently undergoing an fsck after multiple times of going into read-only mode. This is effecting the following guests (Private Servers):

adultspaces mal ps10041 ps10237 ps10337 ps10409 ps10479 ps10603 ps10743 ps10927 ps11032 ps11131 ps11186 ps11563 ps13078 ps14086 ps14479 ps14491 ps16916 ps18608 ps19945 ps22145 ps22196 ps25820 ps26565 ps29915 ps30914 ps4214 ps7899 ps7900 ps7903 ps7919 ps7926 ps7927 ps7972 ps7982 ps8001 ps8101 ps8102 ps8164 ps8293 ps8296 ps8303 ps8403 ps8432 ps8745 ps8830 ps9758 ps9771 ps9996

The fsck takes anywhere from 2-5 hours to complete depending on the amount of data on the server. This server doesn’t have a lot of used disk space so I estimate under 3 hours.

UPDATE 12:40pm PST: This has been fixed at 10:55am, and we’ve been monitoring the server to make sure everything came up correctly. If you’re still having issues, they are most likely unrelated, so please submit a support ticket through the panel with the details: Support/Contact Support.
We apologize for the inconvenience this has caused.


2:32 pm

scheduled maintenance for shared server ‘batman’

Posted (December 21st, 2010 at 2:32 pm PST) by jasony

There will be scheduled maintenance for shared server ‘batman’ tonight at 8pm PST.  We noticed a few days ago that the server started showing filesystem errors and to avoid any further issues, we decided to set up new hardware for it.  We have already done the first rsync of all the current data and tonight we will be replacing the hardware and then running the final rsync of the data.  Once we have services back up and running, we will post an update!  If you have any questions, please contact our friendly support staff!

Update: This has been postponed until either tomorrow morning or tomorrow night.  We will give an update once we have a more definite time frame.

Update: maintenance will start tomorrow, Dec 27th 8pm PST We will be sure to give an update once we have the new hardware up and running.  If you have any questions, please feel free to contact our support staff!

Update Dec 27 8:30pm PST: The new hardware is up and running and services are restoring.  Once services finish restoring, we will start the final rsync of data and then give the final update once it finishes!  If you have any questions or having any trouble, please contact our support staff!

Update Dec 28 2:25pm PST: The final rsync has finished and all services are running normally.  If you are having any trouble or have any questions, please contact our support staff!


1:40 pm

Shared web server ‘buccaneers’ is down

Posted (December 21st, 2010 at 1:40 pm PST) by andrea

We are looking into the cause of the downtime on buccaneers, and will update this post as soon as we know more. We apologize for the inconvenience this is causing.

UPDATE 2:30pm PST: The server is now backup after one of its bad drives was fixed. Your sites should be loading correctly again, but do please let support know if you need further help: Support/Contact Support in panel or http://www.dreamhost.com/contact.cgi .


7:19 am

Shared Server ‘lilongwe’ downtime due to RAID issues **RESOLVED**

Posted (December 21st, 2010 at 7:19 am PST) by Ken

Shared server lilongwe is having a raid issue and is not mounting /home. One of our admins is on the way to our LAX datacenter to fix it. Please be patient with us while we resolve this issue.

Update: 12-21-2010, 09:25 AM PST: The failure was due to two drives having issues within hours of each other. We’ll have to restore from backups as the user partition is no longer readable but the server should be back up within an hour on new hardware. The restore depends on how much data there is total and the rate of transfer which we’re able to sustain (it could be around a week to complete that). We’ll update this post again when the machine is back up.

Update: 12-21-2010, 10:45 AM PST: The web services on the machine are coming up right now and should be complete within the next 30 mins or less. When that’s done, the sites should start loading again (showing data from last available backup), but access may be slow due to the data sync processes running to restore the server to new hardware. We’ll update this post again as soon as we have a better idea how long the sync process will take. In the meantime, feel free to update your sites, as the sync process will not overwrite new data.
If you need help, please contact us with the details either through the panel (Support/Contact Support) or through the contact from at http://www.dreamhost.com/contact.cgi
We apologize for the inconvenience this may be causing, and appreciate your patience during the process.

Update: 12-21-2010, 1:35 PM PST: We are manually reconfiguring the web services right now, and so far so good. The restore is under way now, and the machine load is at good levels. We’re continuing to keep an eye on this, but please let us know if there is anything we can do for you during this process. Please note, if your data was added after the last backup was ran, you will have to upload them again. Sorry about that.

Update: 12-21-2010, 6:10 PM PST: The restore is now 5% done, but in general it doesn’t seem to affect the server much as it’s been holding pretty steady with minimal spikes since the problem. We’re keeping an eye on it to keep to spikes indeed minimal, so the whole process will seem all but seamless. In the meantime, please contact support if you need assistance other than the occasional slow access. Thanks for your continued patience.

Update: 12-21-2010, 8:00PM PST: Bad news :( The new hardware is acting up, so we’re moving the server and data to another new machine. The admins are working on this, and we’ll update as soon we have news on the new machine’s setup. Very sorry about the inconvenience this is causing.

Update: 12-21-2010, 11PM PST: The new hardware setup is now complete: the web services are configured and the latest backups (the same one the previous new hardware was using, from 1am today) are accessible and are showing on your websites. The restore is again running.
We’ll update this post again with our progress. If you need help, please contact support through the panel or at http://www.dreamhost.com/contact.cgi .

Update: 12-22-2010, 11:25am PST:The restore is past the 10% mark and chugging along, we no issues and relatively low loads. We’re continuing to keep an eye on this and will update this post with further progress reports. Thanks for your continued patience.

UPDATE Dec 27 6:45pm PST: The restore on shared server lilongwe has completed. If you continue to experience issues with your services on lilongwe, please contact technical support through the normal channels.

Thank you for your continued patience and support while we strive to continue to improve your DreamHost experience.


2:34 pm

scheduled maintenance for shared server ‘riddler’

Posted (December 17th, 2010 at 2:34 pm PST) by jasony

There will be scheduled maintenance for shared server ‘riddler’ tonight at 8pm PST.  We noticed a few days ago that the server started showing filesystem errors and to avoid any further issues, we decided to set up new hardware for it.  We have already done the first rsync of all the current data and tonight we will be replacing the hardware and then running the final rsync of the data.  Once we have services back up and running, we will post an update!  If you have any questions, please contact our friendly support staff!

Update 8:20pm PST: The new hardware is up and running now and services are restoring.  We will be doing the final rsync once services are completely restored and will provide a final update after the last rsync finishes.  If you are having any issues with your website, please contact our support staff!

Update: The  last rsync has completed!  If you are having any trouble with your website, please contact our support staff!


 
© 1996-2010, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
5:14 am

Shared server achernar is down

Posted (December 17th, 2010 at 5:14 am PST) by Ken

Shared server achernar decided to go to sleep. Our super techs are on the way to LAX to resuscitate it. At this time, no user data should be affected.

We will update this status post as soon as we have more information.

Update 6:30AM PST: The server is back after a hardware swap. If you are on the server achernar and are still experiencing issues, please contact support.


11:23 am

Reduced Download Speeds

Posted (December 15th, 2010 at 11:23 am PST) by Sandon

We have noticed some performance issues with one of our transit providers that provide us connectivity to the internet. Speeds are significantly lower than usual but latency appears to have stayed the same. We are contacting the transit provider to troubleshoot the issue and will post updates.

The reduced performance is not bad enough where it should effect regular browsing of web-sites but might be noticeable on large file downloads. Upload speeds to our network appear to be unaffected.

UPDATE 6:00pm PST: The ticket we opened with the provider (Comcast) is being looked into, and we’re waiting to hear back from them. We’ll update this post again as soon as we get news.

UPDATE 7:23pm PST: We have taken Comcast out of the equation for now. This means that our other providers are currently picking up the slack – and they seem to be doing a pretty good job. Both Comcast’s fiber techs and ours are cleaning our respective ends. Once that’s done, we’ll turn up the link for a few minutes to see if the problems are persisting. Hopefully they don’t. If they do, however, expect another update here.

UPDATE 12:24 AM PST: It looks like the problems was caused by some dirt/dust. After the fiber was cleaned on both ends and the connection brought back up everything is back to normal. We are back to pushing traffic over comcast and not seeing any bad performance like we were earlier. This issue should now be resolved.


8:47 pm

possible interruptions on shared server ‘provenzano’

Posted (December 14th, 2010 at 8:47 pm PST) by jasony

We are going to be making some upgrades to the shared server ‘provenzano’ so you may encounter some interruptions.  We will be sure to update this post once everything is completely done.  If you have any questions at all, please contact our support staff!

UPDATE: Upgrades are finished and services are running smoothly.  If you are having any issues with your website, please contact our support staff!


10:03 pm

Shared server ‘peppy’ downtime *UPDATE 2*

Posted (December 13th, 2010 at 10:03 pm PST) by jasony

It looks like peppy didn’t want to come back up after a reboot.  We have an admin headed to our lax datacenter to check it out and we will be sure to post back as soon as we have an update!  If you have any questions, please contact our support staff!

Update:  The main system partition on peppy took a dive, but all of the customer data in the /home partition was left unharmed.  We’ve reimaged the machine with a fresh operating system, and the apache services are being configured.  We apologize for the downtime, but sites on peppy should be returning bit by bit over the next hour or so.  Thanks for your patience!

UPDATE 14Dec 05:00 PST: Peppy has returned and services restored. If you still experience problems on your shared server peppy, please contact TechSupport through the normal channels.:

UPDATE 14Dec 01:00pm PST: Unfortunately, we’re having issues with Peppy again. The machine was rebooted, and we’re looking into fixing the web services on it. We’ll post again soon with our progress. Apologies for the inconvenience this is causing.

UPDATE 14Dec 01:20pm PST: It looks like we have to reboot the machine again and check the file system to make sure there has been no corruptions. That should take a couple of hours starting about now. We’ll update this post again as soon as there is any news. We appreciate your patience and understanding during this process.

UPDATE 14Dec 03:50pm PST: Unfortunately, due to the large amount of data, the file system check is still going, and at this point we don’t really have an estimate how long it may take. Please check back often for further updates.

UPDATE 14Dec 08:00pm PST: Looks like the file system check failed, so we’re moving this server over to new hardware. We’re hoping to have links up to the backup data, while the fresh server is repopulated (access might be slow, but it will work in the interim).  Sorry about this!

UPDATE 15Dec 11:00am PST: The restore is under way, and from the current progress we estimate that it will finish in less than a week. We’ll update this post if that changes. In the meantime, it’s now running off of the last backups (which is the same data we’re restoring to the new server). As noted above, access is possible, but slow due to the ongoing syncing processes. Feel free to update your data during this process, as it won’t get overwritten by the restore. We appreciate your patience and understanding during this process.

UPDATE 16Dec 2:30pm PST: Just checking in to let you know the restore is almost 30% finished. Again, please accept our sincere apologies the slowness has been causing due to the syncing processes. We’ll update again later with the progress.

Just a quick note to clarify in case it was missed in previous updates: the sites are up and running, but just from backups. If you have any other issues than the slowness, please contact support so they can assist you. What takes the long time is to transfer the data to a new machine where it can start running again locally.

UPDATE 19Dec 1:15pm PST: The restore is now 84% finished. Again, if you’re having issues with your sites other than the slow access due to the load, please contact support with the details.

UPDATE 20Dec 11:40am PST: The restore is almost 95% finished, and the server has been running on a low load for a while now. As always, please direct your questions and requests to our support team for prompt assistance. You can reach us from the panel under Support/Contact Support. If you don’t have access to the panel submit your question at http://www.dreamhost.com/contact.cgi .

Update 21 Dec 10:27pm PST: Looks like the restore has finished!  If you are having any issues or have any questions, please contact our support staff!


6:20 pm

Panel Change: One Click Installs – previously installed software temporarily unavailable

Posted (December 7th, 2010 at 6:20 pm PST) by Ralph

Please note: We’re currently migrating your data to a new-and-improved version of our one-click software installer. If any of your previously-installed software is not appearing on this page, please give it 24 hours and everything will be groovy!


2:21 am

Shared server ‘snowbird’ downtime due to RAID failure

Posted (December 6th, 2010 at 2:21 am PST) by patrickm

We apologize for the lack of a post on this issue, which was identified last night.  The RAID for this server failed and we are in the process of moving all accounts to new hardware.  The process is moving along and we hope to have the server operational again shortly!  We will update this post when the process has been completed and if we have any additional information to provide.  Please feel free to contact Support with any questions you have!

Update – 12pm (PST): We are still working on fully restoring services to this server. Some of the apaches are not getting the proper config files, which is being slowed down by a high load while the data copies over to the new hardware. Our admins are still trying to get all the kinks worked out and to get this server back up to speed. We will update this post once a resolution is near.

Update 1pm All apache processes are running. We have also taken measures to improve i/o performance on the backup machine for the duration of the restore process.

Update Dec 10 5:24pm PST: It appears that snowbird is showing some filesystem errors and needs to undergo a filesystem check.  We are actively monitoring the progress and will post an update once it completes

Update Dec 11, 1:24am PST: That Snowbird, she’s a back up and chirping folks!  Filesystem check completed successfully.   Hopefully she stays aloft for some time to come.  Very sorry about the downtime!

Update Dec 13 2:35pm PST: To ensure snowbird is A-OK, we have rsynced the data to a new server that has no trouble history.  We are swapping hardware now and will post an update once we have it up and running.  We will also then set up the old hardware to do a final rsync of data in hopes you do not have to re-upload a bunch of data you already have!  If you have any questions, please contact our support staff!

Update Dec 13 3:00pm PST: New hardware up, services are restoring now.  After all services are restored, I’ll go ahead and run the final rsync and give a final update once the rsync finishes.  If you are having any issues or have any questions, please don’t hesitate to contact our friendly support staff!

Update Dec 13 6:35pm PST: The final rsync has completed and all services are running normally.  If you are having any issues whatsoever, please contact our support staff!


3:36 pm

Service outages for a handful of servers

Posted (December 5th, 2010 at 3:36 pm PST) by danwy

We are currently experiencing an outage for a handful of servers in one of our datacenters. It has been a recurring issue, but we think that we’re closer to having a solution. Our network admin is in the datacenter now and is working on the situation. We will post an update as soon as we have it. Here is a list of affected servers

acamar achernar achird acrab acubens adhafera adhara aladfar andromeda antila apus aquarius aquila ara arcturus auriga baham becrux beid bellatrix benetnasch betelgeuse betria bharani biham birdun bootes botein brachium bullhead bunda caelum camelopardalis cancer canis canopus capella capricornus carina cassiopeia castor cebalrai celaeno centaurus cepheus chamaeleon chara cheleb chertan chort circinus columba corcaroli crater crux cursa cygnus dabih decrux delphinus deneb dogfish dorado draco equuleus eridanus fornax fresca fringefin gemini grus horologium hydra indus lacerta lepus libra lynx masq megamouth mensa microscopium monoceros norma octans odwalla ophiuchus pavo pegasus perseus phoenix pictor pisces puppis pyxis reticulum sagitta sandtiger scorpius sculptor scutum serpens sextans sprockle taurus telescopium triangulum tucana ursamajor ursaminor vela virgo

UPDATE 5:50pm PST: We changed some of the network hardware and are in the process of reconfiguring the affected system. We appreciate your continued understanding and patience, and will post back as soon as we have more news.

UPDATE 7pm PST: Everything is now back in order. We have been monitoring the system for the past half an hour, and there is no relapse. All sites that we spot checked are loading fine, so we are considering this case resolved for now, but will continue monitoring the network status of these servers.
If you’re still having issues with your domains it’s likely unrelated, so please go ahead and submit a support ticket through the panel. We apologize again for the inconvenience this has caused.


6:03 pm

bunda moving to new hardware

Posted (December 3rd, 2010 at 6:03 pm PST) by jasony

bunda had a raid failure due to 2  mirroring drives dying which requires us to move the server to new hardware and restore from backups.  The last day or 2 of files may need to be re-uploaded.  We will keep everyone posted as things progress!  If you have any questions, please contact our support staff!

Update 10:55 PM PST: There were a few complications getting the new machine up and we apologize for the delay. Bunda is back up and running with all services restored. Service might be a little slower than usual for the next few days while data is still being served off the backup server via NAS and restores are going. Once data is moved back to the new local raid array site speed should return to normal.

Update Dec 7th 03:22pm PST: The restore process has finished and everything should be running fine now.  If you are missing any data or having any issues with your site, please contact our friendly support staff!


4:51 am

Belmopan Downtime

Posted (December 3rd, 2010 at 4:51 am PST) by Sandon

One of our web-servers ‘Belmopan’ currently down. The machine immediately turns itself off after being powered on for a few second and gets caught in that loop suggesting a bad motherboard or possibly PSU. We are moving the disks and raid controller to a new chassis and dont expect any data loss. ETA is ~30 minutes before its back up.

Update 05:19 AM PST: The machine has been moved to new hardware and booted up. It just needs a few configuration changes for the new MAC addresses on the NICs and should be back up within 5 minutes.

Update 05:24 AM PST: And the machine is back up and running and all services restored. Please contact support if your web-server is ‘belmopan’ and your sites are still down.


8:07 am

Homie Cluster Networking Issues

Posted (December 2nd, 2010 at 8:07 am PST) by Justin@DH

We are currently experiencing an issue with a specific row of racked servers in our LAX datacenter. We are looking into the cause but believe it could be DDOS related. We have our network admins on the job already and we will be finding and stopping the cause. We will update this status post when we have more info. The affected servers are:

avarua bangkok ben-gali berbils bern bertha bolkens caracas cardiff cayenne cheetara claudus colombo daggernose damascus dodoma dometome dushanbe funafuti funchal gibraltar hachiman hagatna hanoi harare jackalman jaga jagara khartoum lamnidae leah lion-o lome lusaka lynx-o mamoudzou mandora mbabane monkian montevideo mumm-ra nayda noumea nuuk oranjestad pagopago panthro papeete paramaribo portvila pumyra sainthelier sanaa slithe smoothhound snowman speartooth spurdog stanley stockholm tabbuts taipei tashkent thunderkittens torr torshavn turmagar tygra vaticancity vultureman washingtondc willa wizz-ra wollow

Update: 08:43 AM PST: As of approximately 10 minutes ago the target was found and the destination IP null-routed. The target was on the web-server ‘wollow’ so a few of our customers on the same apache instance as the target (about 5% of the customers on wollow) will still have some downtime until it is renumbered and DNS propagates.


 
© 1996-2010, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
8:23 pm

Issues for our Homie Cluster customers.

Posted (December 1st, 2010 at 8:23 pm PST) by brettg

Earlier this evening we experienced an outage for some of our “homie” cluster users. This issue was related to a semaphore limits on the machines themselves and not all users were affected; we’ve fixed this and are currently investigating the root cause of the semaphore issue. Things should be clear for now but if you do have any questions please contact our support staff via the Panel.


6:25 am

Changes made in Web Panel will be delayed

Posted (December 1st, 2010 at 6:25 am PST) by patrickm

We sincerely apologize for this, as a large number of WordPress automatic upgrades were pushed out last night, which has caused something of a traffic jam for customer updates to services in the Web Panel.  Tasks such as installing new OneClick Installs, changing passwords, adding users, modifying hosting settings on domains and just about anything else that requires a back-end change will be delayed.

Our Admin team is aware of the issue and looking in to a solution, but we may be forced to wait out the updates for this issue to be resolved.  We’ll be investigating why this happened and how to prevent it in the future, as well!  If you have any concerns, or questions, please contact Support for more info!

UPDATE 10:50am PST: The number of delayed updates are slowly but surely decreasing. Currently, we’re estimating a four hour delay, but we’ll update this post again with any progress. Thanks for your continued patience.

UPDATE 3:14pm PST: Unfortunately, we way underestimated the delay, and the new estimate is sometime around 32 hrs. We apologize for the continued inconvenience, and suggest that you keep checking this page for more updates to come.

UPDATE 3:45pm PST 12/2/2010: We’re happy to report that that delay is much smaller than it used to be and continues to decrease. It appears that panel-initiated updates are now going through with minimal wait time. We’re going to keep this status open until we can absolutely make sure that it all caught up.


2:13 am

Spike MySQL down – RESOLVED

Posted (December 1st, 2010 at 2:13 am PST) by JeffH

Disks went into read-only earlier tonight, and raid array failed to rebuild.  We’re failing it over to newer, faster, better hardware.  If your MySQL server is not “spike”, this won’t affect you.  We’ll update as the failover progresses.

Update 5:47am PDT: This hardware replacement has been completed for this shared MySQL server!  If you are still experiencing any issues with this service, please contact Support and they will be glad to help!


2:26 pm

Angel is down

Posted (November 30th, 2010 at 2:26 pm PST) by brians

The machine was not responding to reboots. A member of our admin team is in the process of re-imaging the machine. Everything will be resolved shortly. We sincerely apologize for any inconvenience the downtime had caused and appreciate your continued patience. Please check back for future updates.

UPDATE: 2010-11-30 4:01 PM PST

The machine is back online. Again, we truly appreciate you bearing with us. If you are still having problems, please contact support. It’s likely not related to this outage.


7:42 am

Shared Server ‘bolkens’ downtime due to a Denial of Service Attack

Posted (November 30th, 2010 at 7:42 am PST) by patrickm

We apologize for the downtime our customers on the shared server ‘bolkens’ are experiencing at this time.  A Denial of Service Attack is underway on a domain hosted on this shared server.  Our Admin team is working hard to resolve this issue and restore service for the rest of the server.  We will update this space with more details, should they become available.  If you have any questions, please feel free to contact Support for more information!

The DDoS target has been identified (a domain named after an American company under a Chinese-owned account), and the Apache service has been given a new IP address (those with unique ip addresses would be unaffected by this).


1:54 am

Homie-vserver53 is filesystem checking

Posted (November 30th, 2010 at 1:54 am PST) by JeffH

Homie-vserver53 came back up as read-only after a reboot.  Currently it’s doing a filesystem check, and during that time VPSs will be down.  We apologize for any inconvenience.  We will post updates as they are available.

Update: The filesystem check has been completed for some time now, and vserver guests are now back up.


9:46 pm

terranova moving to new hardware

Posted (November 29th, 2010 at 9:46 pm PST) by jasony

terranova will be moving to new hardware tonight as we’ve been rsyncing its current data to new hardware for the last few days and its just about done.  The last thing that needs to be done is a final rsync which does require web services to be turned off temporarily.  Once the rsync is done, the hardware will be replaced and services will be restored shortly after.  The reason we are doing this is because we noticed several issues with the filesystem and want to avoid having things possibly get worse.  We will post back once things are back up in working order!  If you have any questions, please contact our support staff!

Update 11-30-10 12:57am PST: terranova is now running on shiny new hardware! Yay!  Services are returning and everything is looking good.  If you are having any issues with your site or missing any data, please be sure to contact our friendly support staff!


4:20 pm

homie-vserver07 fscking

Posted (November 29th, 2010 at 4:20 pm PST) by jasony

Well folks, it looks like homie-vserver07 decided to go into a fsck right after a reboot.   The following private servers are affected:

ps11283 ps11550 ps11574 ps11944 ps11996 ps12637 ps12968 ps13010 ps13076 ps18573 ps19523 ps20112 ps21009 ps22766 ps28537 ps35179 ps36030 ps38104 ps38105 ps38159 ps38161 ps38163 ps4431 ps4434 ps4446 ps4449 ps4451 ps4453 ps4455 ps4458 ps4459 ps4674 ps4682 ps4812 ps4828 ps4856 ps4864 ps4922 ps4933 ps5015 ps5047 ps5052 ps5065 ps5067 ps5096 ps5101 ps5108 ps6784 ps8430 ps8443

As soon as the fsck finishes, we will be sure to update this post!  If you have any questions, please contact our friendly support staff!

Update 5:24pm PST: The fsck has finished and the server is now back up and running.  If you are having any issues with your private server mentioned above, please contact our support staff!


9:46 am

Isolated intermittent network issues at LAX

Posted (November 29th, 2010 at 9:46 am PST) by douglas

One of our rows at LAX is experiencing “network flapping” at it’s distribution switch causing intermittent connectivity issues. We are investigating.

Affected machines in H row:

acamar achernar achird acrab acubens adhafera adhara aladfar andromeda antila apus aquarius aquila ara arcturus arizona auriga baham becrux beid bellatrix benetnasch betelgeuse betria bharani biham birdun bootes botein brachium bullhead bunda caelum camelopardalis cancer canis canopus capella capricornus carina cassiopeia castor cebalrai celaeno centaurus cepheus chai chamaeleon chara cheleb chertan chort circinus coffee columba corcaroli crater crush crux crystallight cursa cygnus dabih decrux delphinus deneb dogfish dorado draco equuleus eridanus fornax fresca fringefin gemini grus hi-c horologium hydra indus koolaid lacerta lepus libra lynx masq megamouth mensa microscopium minutemaid monoceros monster mountaindew mrpibb musclemilk norma octans odwalla oj ophiuchus pavo pegasus perseus phoenix pictor pisces punch puppis pyxis reticulum sagitta sandtiger scorpius sculptor scutum serpens sextans silk spacey-vserver08 spacey-vserver11 spacey-vserver12 spacey-vserver13 spacey-vsql02 spacey-vsql03 spaceymail-a1 spaceymail-a2 spaceymail-a3 spaceymail-a4 spaceymail-a6 spaceymail-a7 spaceymail-master spaceymail-mx1 spaceymail-mx2 sprockle tab tang taurus telescopium triangulum tucana ursamajor ursaminor v8 vela virgo

spacey-* machines should all be offloaded at this point.

Update: Normality as returned. You may now return to your boring life.

Something similar happened on Sunday at the same time, our network engineer will be at our LAX data center in case this crops up to isolate the issue to network traffic patterns or failing optical interfaces (laser diodes and LEDs burn out over time).


7:13 am

Shared Server ‘bcrich’ Hardware Failure

Posted (November 29th, 2010 at 7:13 am PST) by patrickm

We apologize for the inconvenience that this hardware failure is causing.  Our team is replacing the hardware for this server and are looking to swap its drives into a new chassis so as to avoid a lengthy restore from backups.  We will update this post when more information is available!

Update: Hardware swap out complete and machine is booting normally with networking.


 
© 1996-2010, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
8:53 am

Achird inodes at 100%

Posted (November 26th, 2010 at 8:53 am PST) by phiya

We’re actively working on freeing up inodes. Services should be operational very soon. I’ll post an update as soon as we’ve sorted the issue out. Sorry for the inconvenience.

Update: 10:00am

Over a million inodes have been cleared up, all services are now back to normal. Again, we do apologize for the inconvenience! (Make sure when you use page caching where it makes sense, and your page cache doesn’t generate new cache files for every single hit on your website without deleting the old ones).


7:58 am

Pretoria hit with UDP flood attack

Posted (November 26th, 2010 at 7:58 am PST) by phiya

We are working to block the attack and restore service as soon as possible. I will provide further updates as they become available. Please check back. We apologize for the inconvenience this has caused all of you.

Update: 10am

The Offending IP addresses have been blocked, and services came back up about an hour ago. Unfortunately, we were also forced to change the IP of the apache instance, which means that some users may have trouble getting to the site while DNS updates(usually takes about 1-4 hours). Clearing out DNS cache in your computer should help flush out the old apache’s IP. Feel free to contact support should further you experience further issues.

Again, we apologize for the inconvenience this has caused all of you.


5:17 pm

Web server sanaa downtime due to technical issues

Posted (November 23rd, 2010 at 5:17 pm PST) by andrea

Web server sanaa was having technical issues, and we found it needs a new raid card. It’s already being tended to, and we estimate the downtime to be around an hour. Sorry about the inconvenience this is causing. We’ll post updates as soon as possible.

UPDATE 6:30PM PST: This is still being worked on, thanks for your patience!

Update 6:43: We’ve replaced the RAID card and the machine should be good to go, please contact our support staff if you notice any further issues.


10:00 pm

Network Maintenance:: adding IPv6 to LAX transit tonight 11/19/2010 22:00 – 00:00

Posted (November 19th, 2010 at 10:00 pm PST) by robertn

Note:  This maintenance was completed.  I will continue to monitor our traffic patterns, but there was no disruption during this window.

We will be adding IPv6 transit onto our AboveNet BGP session on the LAX border router tonight.

While this should have no impact on customer connectivity, all border and core changes have innate risk.  We urge you to plan accordingly.

Posted 11/19/2010 at 15:16


8:00 pm

oslo is getting a fsck!

Posted (November 19th, 2010 at 8:00 pm PST) by jasony

oslo just got one of its disks replaced and then decided to go into ‘read-only filesystem’ mode.  To make sure we have a happy filesystem, oslo is undergoing a filesystem check!  We will be sure to post new updates as things progress!  If you have any questions, please email our friendly support staff!

Update 1:40 AM PST: Oslo is back up and running properly. If you are on oslo and still having problems accessing your site please contact support.


6:03 pm

escobar read-only filesystem

Posted (November 19th, 2010 at 6:03 pm PST) by jasony

If you have been trying to update your site on ‘escobar’, you may have noticed some ‘read-only filesystem’ errors (your site should still load, you just shouldn’t be able to make any changes).  This is because one of the disks died and needed to be replaced and upon replacing it, the mirroring drive decided to take a dump.  After a couple reboots, we were able to get the server back up so the first drive can rebuild.  The rebuild is actually going rather quickly as it’s currently at 41% as I type this.  As soon as that is done, we are going to replace the second drive and wait for that to rebuild.  If all goes as planned, we’ll have a happy server again!   We will keep everyone updated as things progress.  If you have any questions, please feel free to contact our friendly support team!

Update 8:23pm PST: We were able to get the server back up and running and will be monitoring ‘escobar’ extremely close to ensure it stays up.  If you are still noticing issues, please contact our support staff!

Update 11/21 12:25pm PST: Okay so it looks like escobar did go back into read-only mode. We will be moving escobar to new hardware and restoring from backups.  Once that is done, we will rsync any possible remaining data from the old machine to the new.  We will post updates as things progress!  If you have any questions, please contact our support staff!

Update 11/21 03:50pm PST: escobar is now running on new hardware and is having data restored via backups.  We will post back as soon as this process has finished!

Update 11/21 04:30pm PST: We don’t have an estimate yet on how long the restore will take. Feel free to update your files in the meantime if needed, as the restore won’t overwrite your new changes. Please contact support if you need help with your sites and users in the meantime.

Update 11/21 04:30pm PST: Just a quick note to let you know that the restore is coming along. It’s a large amount of data, so it will take a while, but we’ll update you here as soon as there is more progress.

Update 11/27 02:26pm PST: The restore process has finished and everything should be running normally now.  If you are missing any data or having issues with your website loading, please contact our support staff!


9:51 am

PHP errors related to uploading files

Posted (November 18th, 2010 at 9:51 am PST) by Justin@DH

We are looking into an issue where uploading files via php is erroring or resulting in internal server errors. The admins and developers are on the case and when we have further info we will update this post.

UPDATE: An upgrade to the mod_fcgid package seems to be the cause, and we are currently reverting the upgrade on all servers to get them back to a normal working state. We will update this post again when it is complete.

UPDATE 2: Our admin team has let us know that 99% of the servers should now have reverted FastCGI packages — and we should be at 100% around 5 PM, PST.  If you are still seeing issues, please contact tech support right away and we can make sure the fix has been pushed and your Apache server has been properly restarted.


11:11 am

Aldhibah experiencing filesystem issues.

Posted (November 17th, 2010 at 11:11 am PST) by mir

We have dispatched a member of our admin team to correct this issue and will update this post as soon as we have additional details for you. The good news is that the affected filesystem does not contain customer data (thus we expect that we will not have to restore from backups which can be a lengthy process).

Update 11:25 Imaging and customer data is intact, we are now configuring the system. We’re into the final stage and should be done soon.

Update 12:25 We’re a little more than half way through apache configuration runs, all other configs are complete. You can log into the server and update our data files with scp/ssh. Or run “fortune ” as many times as you like from the shell prompt.

Update 2:10 Everything is finished up, if you are still experiencing trouble with one of your sites please contact our support team via the panel!


11:07 pm

homie-vserver129 boot issues

Posted (November 15th, 2010 at 11:07 pm PST) by jasony

homie-vserver129 decided not to come back up after reboot so we are currently investigating the issue and will have it fixed up as soon as we can.  We will be sure to update everyone as things progress!

Update 10:20pm PST: This has been resolved and the machine is back up and running!  Please contact support if you are having any issues at all!


11:02 pm

Minor networking issues with a handful of machines

Posted (November 15th, 2010 at 11:02 pm PST) by Jason C

Machines across two racks in our datacenter had some connectivity issues for a brief time this evening. While a junior admin was investigating the issue, things managed to correct themselves – but that doesn’t mean we’re marking this issue as resolved. Admins are continuing to work on determining the cause of this and will do what they can to make sure the servers on the affected racks stay accessible.

If you happen to notice any issues with your sites, please don’t hesitate to contact support right away.


 
© 1996-2010, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
4:52 pm

FTP connectivity problems

Posted (November 14th, 2010 at 4:52 pm PST) by Jason C

If you can’t currently connect to your FTP account, don’t panic! There was a minor configuration change on our network that was recently pushed out to a batch of servers. Unfortunately, that change is making a number of FTP daemons refuse connections.

We’re really sorry for any issues that this might be causing folks. Our admin team is already aware of the issue and is working on fixing things up right now.

If you need to get files posted to your hosting account and are currently affected by this issue, please consider configuring your user for SFTP access and using that for now.

Update: We’ve resolved the issue for our users, it looks like the cause was due to an attack on port 21. If you’re still experiencing issues please contact our support staff.


8:15 pm

jbplayer raid card issues

Posted (November 11th, 2010 at 8:15 pm PST) by douglas

jbplayer’s raid card is having some issues. There should be no data loss, but we are working to get this resolved.

Update: Raid card replacement is complete, and httpd services are coming back up now.


4:58 pm

Rails 3.0.1 Upgrade

Posted (November 9th, 2010 at 4:58 pm PST) by brettg

Over the course of this month and the next we will be deploying a new  version of Ruby on Rails (3.0.1) along with new versions of RubyGems  and Rake  to help facilitate the ever changing world of RoR. At this  time, we recommend that all people that are using Ruby on Rails freeze  their Rails applications if they do not want to upgrade to 3.0.1.

You can do so by running the following command in your Rails Application Directroy

rake rails:freeze edge VERSION=2.3.8 && rake rails:update

You can later unfreeze your install by running:

rake rails:unfreeze

To  find out more information about what specifically has changed from  2.3.8 to Rails 3 please review the RoR community announcement:

http://weblog.rubyonrails.org/2010/8/29/rails-3-0-it-s-done

If you have any questions please, feel free to contact our support staff via the panel!
Thanks,Your Happy Dreamhost Upgrading Team


3:50 pm

bunda down for fsck!

Posted (November 7th, 2010 at 3:50 pm PST) by jasony

bunda decided to boot its filesystem into read-only mode today so its currently undergoing a filesystem check. This usually takes 2-5 hours but we will be sure to post back with updates as things progress.  If you have any questions, please contact our friendly support staff!

Update 4:19pm PST: The filesystem check has finished and bunda is back in business!  If you are still experiencing issues, please contact our support staff!


11:52 am

Google mod_pagespeed causing issues

Posted (November 7th, 2010 at 11:52 am PST) by douglas

Update: A new patch released fixed the Serf/0.3.1 issue mentioned below. Feel free to experiment with mod_pagespeed. However, the remove_quotes filter breaks XHTML strict validation.

Well, as many people know, we’re now supporting mod_pagespeed. Awesome yeah!

Unfortunately we’re noticing this is causing issues on our servers, particularity the “serf-0.3.1″ module is resulting in many thousands of connections to the the local apache service saturating it’s connections.. This is a very new development and we will be investigating it, however, problem domains will have mod_pagespeed temporarily disabled until this is resolved. These local hits show up in access.log as the “Serf/0.3.1″ user agent. If you’re seeing a large number of hits from this user agent, turn off mod_pagespeed.


6:00 am

homie-vserver09 being moved Sun 11-07 6am

Posted (November 7th, 2010 at 6:00 am PST) by douglas

Update : The planned move has been completed as planned within the estimated time frame. Nothing to see here, move along, move along.

homie-vserver09 will be moved from it’s temporary rack to it’s more permanent location (replacing the position of the old machine that housed homie-vserver09) sometime between 6:00am-6:30am Sunday Nov 7th. The machine will be down for about 3 minutes, and the individual guests will start up shortly there after. Some guests may take as long as 30 minutes to restart, though most will be completed in half that time (and a few lucky ones will start up a few minutes after the machine boots).

Yes this will be just 3 minutes downtime for the machine (max 30 for all guest to start). Robert N, or network engineer, sent a a bunch of the network gremlins packing, so no need to worry. Which remind me, I need get more fuel for my flame thrower…

(Actually the machine isn’t changing it’s switch, just were it gets power from and swapping out the 30ft network cables with 5ft cables– so the network changes to resolve some long standing core router issues, and odd/undocumented behavior of the current Cisco routers/switch firmware, would have no affect on this move)

The following guests will be affected:

ps11946 ps12015 ps12198 ps12512 ps13158 ps13210 ps13233 ps13933 ps14070 ps14071 ps14385 ps14757 ps16765 ps17605 ps17970 ps18342 ps18354 ps18355 ps19682 ps21033 ps21234 ps21623 ps22752 ps35233 ps35235 ps4484 ps4492 ps4502 ps4507 ps4510 ps4513 ps4529 ps4537 ps4544 ps4807 ps4829 ps4857 ps4927 ps4954 ps5020 ps5698 ps5723 ps6474 ps6527 ps6737 ps6754 ps7422 ps7475 ps7478 ps7479 ps7488 ps7498 ps7947 ps8106


10:00 pm

Network Maintenance: November 5th 22:00 – 02:00

Posted (November 5th, 2010 at 10:00 pm PST) by robertn

This completes the maintenance window.  We have now completed all of our major work on our network infrastructure.

Thank you for your patience and support.

NOTE:  This maintenance window has been extended for another 2 hours (to 04:00).  Please bear with us.

We will be completing the fixes to our Core infrastructure.  This will include abandoning HSRP, restructuring our internal LAN routing domain and pruning unnecessary vlans from our border routers.

Please plan for intermittent outages during this window.

If you experience a prolonged outage, please contact your Dreamhost Technical Services Representative.


5:48 pm

Homie mail maintenance 11/6 11:00PM PDT

Posted (November 3rd, 2010 at 5:48 pm PST) by jordan

On Sunday 11/6 at 11:00PM PDT, we will be temporarily interrupting mail service for customers on homiemail-master. This outage is expected last about 1 hour and will affect the following services: imap,pop,smtp (incoming/outgoing) & webmail. There should be no lost/bounced mail , but there will be a delay in incoming mail due to the outage. The purpose of this upgrade is to convert older courier mail servers to dovecot ( a new and improved imap,pop server ) , which should increase the performance & reliability of these servers. This post will be updated when we start/finish and technical support will available to help any of you in case there is a problem with your account after this conversion.

Update: 11/6 02:47 PM PDT Homie mail maintenance has been postponed due to network related outages this morning.


5:52 am

Server luanda is losing itself

Posted (November 3rd, 2010 at 5:52 am PST) by Ken

The server luanda is having raid card issues. Our admins are actively working on getting the hardware replaced. We will update this post as more information becomes available.

Update 07:20 AM PDT: We got new hardware setup and the users/services are currently being configured. Services should start returning shortly and be all back up within 30 minutes or so. The old array was unfortunately no longer in a usable state and we had to mount the backup server and start restoring from backups. This means data uploaded in the last 24-48 hours will need to be re-uploaded. More updates to come.

Update 08:42 AM PDT: It took longer than expected to get everything configured but all services should be restored. Please contact support if you are still having issues with your site.

Update: To clarify the last update, please note that the restore from backup is still happening on this server.  This means that many accounts will still see their backup data as their live home directory.  This also means that the load on the server is higher than normal, as we are running multiple rsync processes to restore the data as quickly as possible.  We will update this post when the restore process is completely finished.


12:26 pm

Networking problems causing downtime

Posted (November 2nd, 2010 at 12:26 pm PST) by danwy

We’re currently experiencing some networking issues on our end that is causing downtime for websites and MySQL. It started about 15-20 minutes ago and we’re not sure of the cause yet. Our admin team is already investigating and we’ll update this post once we have more information on the cause and what servers/services are affected.

Update – 12:50pm (PST): We’ve narrowed this down to a specific server rack. Our admin team is still investigating the issue, but have passed on a list of affected servers:

*SEE BELOW*

We’ll update this post again when we have more info.

Update – 1:40pm (PST): This issue has turned out to be bigger than we initially suspected. We thought it was just one server rack that was initially affected, but we’ve diagnosed more racks having the same issue. Here is an updated list of servers that may potentially still be having problems (just because it’s on this list doesn’t mean that it’s actually having problems, just that it’s in the affected rack):

Web/MySQL servers:
alamak alaraph alathfar albaldah albali albireo alchiba alcor alcyone aldebaran alderamin aldhafera aldhanab aldhibah aldib alfawaris alfirk algedi algenib algieba algol algorab alhajoth alhena alioth alkaid alkalbalrai alkalurops alkaphrah alkurah alkurud almach alnair avarua bangkok ben-gali berbils bern bertha bolkens caracas cardiff cayenne cheetara claudus colombo daggernose damascus dodoma dometome dushanbe funafuti funchal gibraltar hachiman hagatna hanoi harare jackalman jaga jagara khartoum lamnidae leah lion-o lome lusaka lynx-o malcolm mamoudzou mandora mbabane misha monkian montevideo mumm-ra nayda noumea nuuk oranjestad pagopago panthro papeete paramaribo pontadelgada portvila pumyra sainthelier sanaa slithe smoothhound snowman speartooth spurdog stanley stockholm tabbuts taipei tashkent thunderkittens torr torshavn tunis turmagar tygra vaticancity vultureman washingtondc willa wizz-ra wollows

PS & PSMYSQL servers:
ps27247 ps27248 ps27249 ps27250 ps27271 ps27274 ps27288 ps27299 ps27329 ps27330 ps27339 ps27343 ps27352 ps27353 ps27359 ps27367 ps27378 ps27386 ps27388 ps27389 ps27391 ps27392 ps27393 ps27396 ps27397 ps27398 ps27400 ps27401 ps27403 ps27404 ps27405 ps27407 ps27414 ps27415 ps27416 ps27418 ps27419 ps27423 ps27424 ps27425 ps27426 ps27427 ps27428 ps27429 ps27430 ps27431 ps27432 ps27450 ps27471 ps27472 ps27474 ps27475 ps27476 ps27477 ps27478 ps27479 ps27480 ps27484 ps27490 ps27491 ps27492 ps27493 ps27495 ps27496 ps27497 ps27498 ps27499 ps27500 ps27510 ps27523 ps27526 ps27527 ps27531 ps27532 ps27533 ps27534 ps27535 ps27536 ps27538 ps27543 ps27558 ps27580 ps27588 ps27590 ps27592 ps27604 ps27615 ps27627 ps27648 ps27657 ps27680 ps27703 ps27720 ps27723 ps27736 ps27741 ps27749 ps27765 ps27769 ps27784 ps27799 ps27831 ps27864 ps27884 ps27940 ps27951 ps27961 ps28015 ps28032 ps28067 ps28105 ps28258 ps28265 ps28305 ps28307 ps28310 ps28311 ps28536 ps28546 ps28554 ps28576 ps28577 ps28578 ps28582 ps28583 ps28584 ps28585 ps28586 ps28606 ps28614 ps28616 ps28937 ps28940 ps28941 ps28942 ps28943 ps28954 ps29076 ps29088 ps29101 ps29104 ps29131 ps29132 ps29135 ps29136 ps29137 ps29139 ps29141 ps29143 ps29144 ps29145 ps29146 ps29147 ps29148 ps29149 ps29150 ps29168 ps29173 ps29182 ps29184 ps29188 ps29238 ps29247 ps29252 ps29253 ps29258 ps29259 ps29261 ps29263 ps29265 ps29266 ps29268 ps29271 ps29272 ps29273 ps29275 ps29316 ps29328 ps29331 ps29357 ps29363 ps29369 ps29376 ps29418 ps29425 ps29426 ps29427 ps29428 ps29429 ps29430 ps29431 ps29432 ps29433 ps29444 ps29455 ps29467 ps29477 ps29481 ps29483 ps29484 ps29485 ps29529 ps29550 ps29557 ps29562 ps29881 ps29896 ps29928 ps30000 ps30005 ps30066 ps30121 ps30157 ps30158 ps30160 ps30161 ps30163 ps30164 ps30168 ps30170 ps30171 ps30172 ps30173 ps30174 ps30175 ps30176 ps30177 ps30179 ps30180 ps30181 ps30182 ps30184 ps30187 ps30190 ps30191 ps30193 ps30194 ps30195 ps30197 ps30198 ps30201 ps30202 ps30203 ps30204 ps30205 ps30206 ps30207 ps30208 ps30212 ps30285 ps30313 ps30329 ps30333 ps30334 ps30339 ps30340 ps30341 ps30342 ps30345 ps30346 ps30348 ps30350 ps30353 ps30354 ps30356 ps30357 ps30358 ps30360 ps30361 ps30362 ps30363 ps30364 ps30365 ps30366 ps30367 ps30368 ps30369 ps30386 ps30394 ps30395 ps30396 ps30397 ps30398 ps30399 ps30400 ps30401 ps30402 ps30403 ps30404 ps30406 ps30407 ps30495 ps30500 ps30501 ps30529 ps30536 ps30539 ps30597 ps30652 ps30665 ps30704 ps30720 ps30727 ps30730 ps30751 ps30785 ps30793 ps30817 ps30837 ps30898 ps30938 ps30948 ps30953 ps30955 ps30959 ps30966 ps30993 ps30999 ps31012 ps31027 ps31031 ps31038 ps31059 ps31082 ps31127 ps31137 ps31144 ps31153 ps31201 ps31313 ps31328 ps31386 ps31391 ps31399 ps31413 ps31417 ps31466 ps31546 ps31572 ps31596 ps31626 ps31650 ps31738 ps31741 ps31752 ps31769 ps31773 ps31815 ps31838 ps31855 ps31871 ps31900 ps31908 ps32007 ps32043 ps32068 ps32092 ps32168 ps32213 ps32262 ps32289 ps32326 ps32338 ps32347 ps32374 ps32385 ps32390 ps32441 ps32446 ps32450 ps32484 ps32506 ps32539 ps32564 ps32592 ps32626 ps32660 ps32684 ps32707 ps32716 ps32732 ps32778 ps32781 ps32790 ps32795 ps32804 ps32815 ps32828 ps32890 ps32958 ps32975 ps32986 ps32998 ps33022 ps33060 ps33097 ps33125 ps33173 ps33184 ps33212 ps33216 ps33238 ps33247 ps33255 ps33271 ps33279 ps33291 ps33293 ps33301 ps33331 ps33350 ps33353 ps33411 ps33438 ps33594 ps33596 ps33597 ps33598 ps33609 ps33846 ps33861 ps33862 ps33863 ps33864 ps33865 ps33867 ps33872 ps33873 ps33874 ps33875 ps33876 ps33877 ps33878 ps33880 ps33881 ps33882 ps33883 ps33886 ps33887 ps33890 ps33892 ps33893 ps33906 ps33938 ps33946 ps33951 ps33969 ps33981 ps33990 ps34014 ps34015 ps34016 ps34017 ps34018 ps34019 ps34020 ps34021 ps34024 ps34025 ps34026 ps34028 ps34031 ps34032 ps34035 ps34043 ps34069 ps34109 ps34168 ps34180 ps34202 ps34204 ps34207 ps34244 ps34274 ps34277 ps34305 ps34325 ps34334 ps34367 ps34372 ps34417 ps34484 ps34485 ps34500 ps34516 ps34517 ps34533 ps34534 ps34535 ps34536 ps34537 ps34538 ps34539 ps34540 ps34542 ps34543 ps34545 ps34546 ps34547 ps34549 ps34550 ps34555 ps34563 ps34569 ps34571 ps34573 ps34575 ps34578 ps34588 ps34590 ps34591 ps34613 ps34614 ps34615 ps34618 ps34622 ps34629 ps34633 ps34646 ps34651 ps34652 ps34653 ps34655 ps34658 ps34659 ps34663 ps34666 ps34668 ps34670 ps34674 ps34676 ps34678 ps34679 ps34680 ps34683 ps34704 ps34706 ps34713 ps34725 ps34744 ps34765 ps34776 ps34785 ps34800 ps34814 ps34827 ps34828 ps34829 ps34833 ps34881 ps34883 ps34884 ps34885 ps34886 ps34887 ps34888 ps34890 ps34891 ps34892 ps34893 ps34894 ps34896 ps34897 ps34901 ps34935 ps34943 ps34963 ps34967 ps34970 ps34978 ps34979 ps34981 ps34983 ps35013 ps35014 ps35018 ps35048 ps35069 ps35071 ps35087 ps35100 ps35121 ps35126 ps35130 ps35135 ps35138 ps35139 ps35144 ps35146 ps35147 ps35148 ps35149 ps35156 ps35166 ps35174 ps35193 ps35196 ps35212 ps35226 ps35238 ps35250 ps35253 ps35288 ps35292 ps35307 ps35309 ps35314 ps35320 ps35348 ps35355 ps35364 ps35368 ps35378 ps35385 ps35387 ps35388 ps35389 ps35392 ps35401 ps35402 ps35417 ps35435 ps35456 ps35512 ps35533 ps35623 ps35630 ps35650 ps35663 ps35678 ps35681 ps35742 ps35747 ps35775 ps35790 ps35793 ps35825 ps35828 ps35848 ps35882 ps35915 ps35919 ps35964 ps35992 ps35996 ps36008 ps36028 ps36060 ps36069 ps36079 ps36095 ps36100 ps36191 ps36202 ps36226 ps36240 ps36246 ps36262 ps36284 ps36285 ps36299 ps36365 ps36379 ps36487 ps36525 ps36544 ps36559 ps36566 ps36632 ps36648 ps36688 ps36690 psmysql27001 psmysql27002 psmysql27011 psmysql27016 psmysql27018 psmysql27019 psmysql27092 psmysql27110 psmysql27155 psmysql27158 psmysql27161 psmysql27167 psmysql27169 psmysql27174 psmysql27199 psmysql27212 psmysql27217 psmysql27241 psmysql27244 psmysql27245 psmysql27265 psmysql27273 psmysql27282 psmysql27287 psmysql27292 psmysql27293 psmysql27297 psmysql27300 psmysql27314 psmysql27319 psmysql27320 psmysql27340 psmysql27351 psmysql27377 psmysql27385 psmysql27395 psmysql27411 psmysql27417 psmysql27434 psmysql27465 psmysql27483 psmysql27516 psmysql27522 psmysql27525 psmysql27556 psmysql27572 psmysql27577 psmysql27609 psmysql27611 psmysql27618 psmysql27622 psmysql27623 psmysql27632 psmysql27634 psmysql27637 psmysql27640 psmysql27642 psmysql27650 psmysql27656 psmysql27666 psmysql27690 psmysql27696 psmysql27697 psmysql27701 psmysql27727 psmysql27732 psmysql27756 psmysql27764 psmysql27770 psmysql27774 psmysql27796 psmysql27803 psmysql27811 psmysql27814 psmysql27824 psmysql27833 psmysql27837 psmysql27841 psmysql27844 psmysql27890 psmysql27901 psmysql27903 psmysql27930 psmysql27933 psmysql27935 psmysql27970 psmysql27973 psmysql27983 psmysql28003 psmysql28014 psmysql28017 psmysql28034 psmysql28050 psmysql28058 psmysql28074 psmysql28079 psmysql28100 psmysql28119 psmysql28141 psmysql28162 psmysql28180 psmysql28213 psmysql28215 psmysql28217 psmysql28242 psmysql28296 psmysql28304 psmysql28329 psmysql28330 psmysql28336 psmysql28363 psmysql28425 psmysql28432 psmysql28463 psmysql28513 psmysql28570 psmysql28575 psmysql28591 psmysql28598 psmysql28617 psmysql28711 psmysql28714 psmysql28735 psmysql28736 psmysql28744 psmysql28774 psmysql28776 psmysql28786 psmysql28805 psmysql28826 psmysql28838 psmysql28841 psmysql28843 psmysql28846 psmysql28857 psmysql28860 psmysql28866 psmysql28870 psmysql28890 psmysql28892 psmysql28899 psmysql28902 psmysql28908 psmysql28915 psmysql28927 psmysql28933 psmysql28938 psmysql28946 psmysql28966 psmysql28972 psmysql28974 psmysql28977 psmysql28986 psmysql28997 psmysql29007 psmysql29014 psmysql29035 psmysql29038 psmysql29044 psmysql29048 psmysql29049 psmysql29068 psmysql29071 psmysql29078 psmysql29111 psmysql29116 psmysql29118 psmysql29128 psmysql29134 psmysql29138 psmysql29165 psmysql29177 psmysql29201 psmysql29205 psmysql29212 psmysql29221 psmysql29222 psmysql29223 psmysql29225 psmysql29232 psmysql29242 psmysql29286 psmysql29293 psmysql29304 psmysql29323 psmysql29346 psmysql29348 psmysql29350 psmysql29449 psmysql29480 psmysql29495 psmysql29519 psmysql29539 psmysql29564 psmysql29568 psmysql29579 psmysql29586 psmysql29592 psmysql29597 psmysql29660 psmysql29688 psmysql29698 psmysql29705 psmysql29711 psmysql29719 psmysql29733 psmysql29737 psmysql29781 psmysql29805 psmysql29806 psmysql29851 psmysql29868 psmysql29880 psmysql29882 psmysql29884 psmysql29889 psmysql29903 psmysql29913 psmysql29964 psmysql29976 psmysql29987 psmysql30020 psmysql30069 psmysql30083 psmysql30084 psmysql30086 psmysql30129 psmysql30132 psmysql30141 psmysql30148 psmysql30186 psmysql30188 psmysql30229 psmysql30260 psmysql30277 psmysql30284 psmysql30287 psmysql30309 psmysql30318 psmysql30335 psmysql30422 psmysql30425 psmysql30504 psmysql30564 psmysql30573 psmysql30576 psmysql30577 psmysql30598 psmysql30601 psmysql30602 psmysql30603 psmysql30604 psmysql30623 psmysql30641 psmysql30651 psmysql30658 psmysql30660 psmysql30672 psmysql30690 psmysql30695 psmysql30729 psmysql30736 psmysql30738 psmysql30755 psmysql30761 psmysql30764 psmysql30767 psmysql30841 psmysql30850 psmysql30894 psmysql30908 psmysql30910 psmysql30911 psmysql30912 psmysql30915 psmysql30916 psmysql30919 psmysql30937 psmysql30939 psmysql30963 psmysql30979 psmysql30983 psmysql30984 psmysql30986 psmysql30990 psmysql30992 psmysql31008 psmysql31010 psmysql31020 psmysql31021 psmysql31058 psmysql31065 psmysql31101 psmysql31108 psmysql31115 psmysql31122 psmysql31186 psmysql31195 psmysql31536 psmysql31780 psmysql32486 psmysql32885 psmysql33058 psmysql33096 psmysql33387 psmysql33421 psmysql33434 psmysql33517 psmysql33573 psmysql33581 psmysql33600 psmysql33622 psmysql33688 psmysql33707 psmysql33732 psmysql33746 psmysql33751 psmysql33755 psmysql33764 psmysql33803 psmysql33836 psmysql33905 psmysql33928 psmysql33929 psmysql33940 psmysql33973 psmysql33995 psmysql34000 psmysql34041 psmysql34048 psmysql34076 psmysql34117 psmysql34169 psmysql34172 psmysql34179 psmysql34227 psmysql34240 psmysql34253 psmysql34276 psmysql34364 psmysql34365 psmysql34366 psmysql34370 psmysql34371 psmysql34373 psmysql34376 psmysql34377 psmysql34379 psmysql34402 psmysql34435 psmysql34441 psmysql34446 psmysql34448 psmysql34459 psmysql34460 psmysql34475 psmysql34477 psmysql34495 psmysql34499 psmysql34501 psmysql34514 psmysql34525 psmysql34530 psmysql34560 psmysql34581 psmysql34596 psmysql34597 psmysql34624 psmysql34661 psmysql34662 psmysql34664 psmysql34665 psmysql34667 psmysql34671 psmysql34677 psmysql34682 psmysql34693 psmysql34700 psmysql34703 psmysql34729 psmysql34731 psmysql34749 psmysql34756 psmysql34768 psmysql34771 psmysql34775 psmysql34779 psmysql34784 psmysql34786 psmysql34787 psmysql34793 psmysql34796 psmysql34799 psmysql34815 psmysql34818 psmysql34824 psmysql34851 psmysql34853 psmysql34860 psmysql34861 psmysql34863 psmysql34866 psmysql34870 psmysql34871 psmysql34878 psmysql34880 psmysql34902 psmysql34906 psmysql34915 psmysql34918 psmysql34921 psmysql34923 psmysql34924 psmysql34929 psmysql34931 psmysql34932 psmysql34937 psmysql34940 psmysql34941 psmysql34942 psmysql34944 psmysql34945 psmysql34946 psmysql34947 psmysql34948 psmysql34949 psmysql34951 psmysql34952 psmysql35002 psmysql35034 psmysql35036 psmysql35052 psmysql35056 psmysql35068 psmysql35083 psmysql35096 psmysql35103 psmysql35104 psmysql35105 psmysql35116 psmysql35123 psmysql35127 psmysql35141 psmysql35178 psmysql35180 psmysql35209 psmysql35219 psmysql35224 psmysql35225 psmysql35230 psmysql35234 psmysql35241 psmysql35245 psmysql35260 psmysql35264 psmysql35278 psmysql35295 psmysql35319 psmysql35356 psmysql35367 psmysql35373 psmysql35390 psmysql35398 psmysql35406 psmysql35410 psmysql35416 psmysql35431 psmysql35437 psmysql35438 psmysql35474 psmysql35478 psmysql35488 psmysql35497 psmysql35498 psmysql35502 psmysql35504 psmysql35513 psmysql35524 psmysql35539 psmysql35552 psmysql35559 psmysql35579 psmysql35593 psmysql35605 psmysql35614 psmysql35625 psmysql35629 psmysql35632 psmysql35667 psmysql35672 psmysql35673 psmysql35676 psmysql35691 psmysql35692 psmysql35693 psmysql35695 psmysql35707 psmysql35751 psmysql35752 psmysql35753 psmysql35754 psmysql35757 psmysql35758 psmysql35764 psmysql35767 psmysql35784 psmysql35801 psmysql35803 psmysql35815 psmysql35826 psmysql35849 psmysql35853 psmysql35878 psmysql35901 psmysql35921 psmysql35942 psmysql35990 psmysql35993 psmysql35994 psmysql35998 psmysql36000 psmysql36013 psmysql36021 psmysql36054 psmysql36081 psmysql36082 psmysql36083 psmysql36086 psmysql36092 psmysql36097 psmysql36102 psmysql36113 psmysql36124 psmysql36144 psmysql36152 psmysql36154 psmysql36187 psmysql36192 psmysql36197 psmysql36198 psmysql36269 psmysql36277 psmysql36293 psmysql36315 psmysql36316 psmysql36329 psmysql36330 psmysql36336 psmysql36345 psmysql36430 psmysql36447 psmysql36463 psmysql36464 psmysql36470 psmysql36472 psmysql36532 psmysql36535 psmysql36542 psmysql36545 psmysql36568 psmysql36607

We are working as hard as we can to get these issues resolved and will post an update as soon as we have any more information.

Update – 2:20pm (PST): This issue should be resolved now. There was a problem with HSRP that affected the switch that the machines above were plugged into. This caused the CPU load on the switch to spike and cause issues on all the servers that connect to it. We’ve confirmed that this issue with the switch has been resolved and we’re in the process of verifying that all affected servers have had their services restored. If you continue to have issues, please contact support. We apologize for the inconvenience that this has caused you.


 
© 1996-2010, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
8:39 am

cardiff being hit by a DDoS

Posted (November 1st, 2010 at 8:39 am PST) by douglas

Evidently a website hosted on cardiff is well liked by someone– so much so they decided to show their love by launched SYN flood DDoS attack against the machine. Since the IP address being hit is a shared, it can take some time to isolate the target of the attack.

Update 9:55am: We have disabled the most probable target of the DDoS attack though we had to assign the apache services a new ip address as a result of it. Since most ISP’s and other servers cache DNS requests, it can take a few hours before the old DNS entries are cleared out and the new IP address is accepted for the domains. So yes, while we have restored the apache service, you may still be getting connection errors for a few hours.


11:00 pm

Network Maintenance :: October 31st, 2010 23:00-02:00

Posted (October 31st, 2010 at 11:00 pm PST) by robertn

Update:  This maintenance was completed within the service window.  The major architectural changes to our network are, with one exception, completed.  I anticipate smoother sailing.

I have read your comments, and will change scheduled maintenance windows to Saturday night/Sunday morning from now on.  While I am an ugly American, I certainly don’t want to be an Ugly American.  I cherish our global customers.

Due to the troubles with our interior routing that caused this weekend’s outages, we will be re-architecting our OSPF domains.  This could cause intermittent outages of 30 seconds to 5 minutes over the course of this window.

By the end of this outage we will have improved network performance, implemented backup routing architecture to avoid issues in the future, and integrated our new border router.

If you notice an extended outage after the end of the window, please contact your Dreamhost Technical Service Representative.


6:08 am

Tirane Hardware Failure

Posted (October 31st, 2010 at 6:08 am PST) by JeffH

Server, Tirane, just took a dive and appears to have suffered major hardware failure (likely the motherboard).  We are in the process of failing over the machine now, and will post updates as they are available.  Thanks for your patience while we get you back up and running.

Update: By way of some devilishly clever troubleshooting by our NOC team (thanks Robert!), we were able to avoid a complete failover, as it was determined that a single faulty hard drive was preventing the entire machine from powering on.  How about ‘dem apples?  In any case, the machine is back up with a new drive in that slot, and the RAID array is rebuilding.   Thanks for your patience!   Phew!


1:07 pm

Networking issues

Posted (October 30th, 2010 at 1:07 pm PST) by mir

We are currently experiencing service interruptions across our entire network in the form of time outs and connectivity problems. We’ve also seen reports of payment processing issues (thanks to failed connections with our upstream providers).

Our networking expert is investigating and we’ll post further details as they become available. We apologize for the trouble this causes.

Update, 13:25 PM PST: We are still experiencing a bit of fallout. Currently one of our data centers is unable to connect to the server which houses dreamhoststatus and one of our name servers (ns2.dreamhost.com) but our LAX facility can. We experienced a complete outage from roughly 12:50 to 13:05.

Our networking expert has narrowed the problem down to changes we were attempting to implement last night and is still working on a complete rollback fix. More updates to follow.

Update, 14:05 PM PST: Our networking expert has rolled back the changes and service should be fully restored.


10:00 pm

New Transit for LAX! New border router for LAX!

Posted (October 29th, 2010 at 10:00 pm PST) by robertn

Update #2 5:50 am Saturday, October 30th:  Our new Border is a little to efficient.  It has disrupted the routing table and is greedily grabbing all of our traffic.  This has lead to a CPU spike and lock up in our cores and disrupted traffic for the last half an hour.  We have rolled back to our previous configuration.

Update:  AboveNet doesn’t seem to be ready.  They are dispatching a technician to check on their connectivity to our Demarc.  Looks like this is an early Halloween trick!

We will be turning up a shiny, new transit link with AboveNet for our LAX data center tonight at 22:00 PDT.  We will also be rolling out a new border router.  This should improve latency for our LAX data center and provide fail-over capacity for our entire network.  We will no longer have our trusty downtown border router carrying the entire load and can eliminate this as a single point of failure scenario from our contingency planning.

This roll out will culminate a year-long initiative to improve our network and increase our capacity.

We anticipate no interruption of service during this maintenance window. But, as with all core and border architectural changes, that possibility exists.  Please plan accordingly.  If you do observe an outage, please contact your Happy Dreamhost Technical Service representative immediately.

Thank you for your support.


12:36 pm

Apache (and site) downtime

Posted (October 29th, 2010 at 12:36 pm PST) by mir

We starting having problems with our apaches exhausting semaphores around 11:00 AM PST today but we were able to clear those out by around noon. Unfortunately we are now seeing another round of affected apache servers and so we are working on cleaning those up as well. We expect to be finished in the next hour or so with that clean up.

Update 15:45 PM PST:

The updates to fix this issue completed about an hour and a half ago – we believe the issue was related to log rotation and will investigate it further to avoid a repeat of this issue. Please contact us if your site are still timing out.


11:00 am

MySQL connections failing

Posted (October 29th, 2010 at 11:00 am PST) by mir

We’re investigating an issue that has causes database connection errors (our servers are using incorrect information to connect to the MySQL servers). A fix is being run now and should complete shortly (within 10-15 minutes), we apologize for this inconvenience.

Update 12:00 PM PST::

The fix completed running (due the volume of machines effected it took longer than expected to complete for which I apologize). We did finish fixing those at around 11 AM PST so everything should be resolved at this time.

Update 14:00 PM PST::

We have found a few instances of this issue effecting Private Servers and are applying the fix to those machines as well (that should be completed in the next hour or two).


4:37 am

Chisinau down for an fsck

Posted (October 28th, 2010 at 4:37 am PST) by Sandon

I had to bring the web-server ‘Chisinau’ down in order to run an fsck on it as the file-system has forced itself into read-only mode do to errors. The fsck can take anywhere from 2-5 hours. Updates to the fsck progress will be written here.

Update 07:12 AM PDT: The server has finished its first pass. This usually means it is approximately 40-50% completed.

Update 12:20 PM PDT: And, it’s back! If you continue to have problems with services on this server, please contact support for further assistance.


3:10 pm

Homie-vserver137 hardware failure

Posted (October 27th, 2010 at 3:10 pm PST) by Leon

Affected VPS:

ps19914 ps19992 ps20052 ps20107 ps20216 ps20383 ps20448 ps20477 ps20480 ps20484 ps20509 ps20559 ps20575 ps20582 ps20748 ps20824 ps20976 ps21143 ps21189 ps21205 ps21298 ps21447 ps21673 ps21836 ps21855 ps21896 ps21912 ps22038 ps22167 ps22478 ps22717 ps22954 ps23178 ps23179 ps23195 ps23761 ps23805 ps23849 ps23887 ps23981 ps24023 ps24045 ps24190 ps24315 ps24389 ps25133 ps25477 ps28072 ps28529 ps28715 ps29034 ps29649 ps29677 ps29696 ps30072 ps30144 ps30283 ps30612 ps30831 ps31000 ps34217 ps34359 ps34515 ps34823 ps34855 ps34882 ps35192

We’re moving all the data to new hardware now. Should be up and running again soon-ish. We’ll update when we have more details.

Update: The server has been back for a couple of hours and we were monitoring it before we posted an update. All should be good; please contact our support staff if you have any questions or concerns.


2:22 am

Network issues with LAX datacenter causing widespread downtime

Posted (October 27th, 2010 at 2:22 am PST) by patrickm

Note:  Wednesday Oct. 27th, 4:04 — Alchemy has recovered our point-to-point port-channel to our LAX data center and corrected the issue.  We have also taken steps to see that this link will recover gracefully and become more robust.

Note:  We have diagnosed this as a problem with our point-to-point link via Alchemy’s fiber.  We have contacted their NOC and  are working closely with them to resolve this issue quickly and permanently.

We apologize for the inconvenience that this outage is causing at this time.  We just lost connection to our LAX data center, which is causing widespread downtime for our customers.  Our network engineer is on-site and working to resolve the problem.  We will update this post when we have more information and when this outage is resolved.  Thank you for your patience.


 
© 1996-2010, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
10:00 pm

Network Maintenance — October 26th, 2010 @ 22:00

Posted (October 26th, 2010 at 10:00 pm PST) by robertn

NOTE:  ended at 1:30 am.  We have successfully diagnosed and corrected our hardware problems on our core switches.

To resolve the sporadic networking issues and investigate the VSS bug we ran up against last week, we have scheduled a network maintenance with our Cisco friends for this evening.  While we hope to avoid any disruption of service, and will take every precaution to minimize downtime for you and your site, this may prove to be unavoidable.

At the conclusion of our session we will fail our core over to the secondary, and take the primary offline for further testing.

The bottom line, you ask?  There is potential for intermittent outages from 22:00 until 02:00.  Please plan accordingly.

Thank you for your support.


5:55 pm

Random bounces in homiemail-sub4 and homiemail-sub5

Posted (October 26th, 2010 at 5:55 pm PST) by andrea

We’re currently experiencing some difficulties with incoming mail (mail is randomly bouncing) on domains using homiemail-sub4 and homiemail-sub5. The admins are working on this, and we’ll update this post as soon as there is progress.

To see if your account may be affected, go to your panel and check your ‘Your Email Server:’ under the Account Status on the upper right-hand corner. We apologize for all inconvenience this is causing.

UPDATE 7:21pm PDT: This issue should now be fixed, our tests show that no mail is bouncing to the users we tested. If you’re still seeing bounces, please let support know!


2:03 pm

Networking issues causing downtime

Posted (October 25th, 2010 at 2:03 pm PST) by danwy

We’re currently experiencing some issues with our networking for a small portion of our servers that is causing customer sites to be unavailable. We’re not quite sure what the cause is yet, but our admins are already on the case and are working on resolving this ASAP. Here is a list of machines in the server rack that may be affected:

Web servers:
avarua bangkok ben-gali berbils bern bertha bolkens caracas cardiff cayenne cheetara claudus colombo damascus dodoma dometome dushanbe funafuti funchal gibraltar hachiman hagatna hanoi harare homie-vserver184 homie-vserver185 homie-vserver186 homie-vserver187 homie-vserver188 homie-vserver190 homie-vserver191 homie-vserver192 homie-vserver193 homie-vserver194 homie-vserver195 jackalman jaga jagara khartoum leah lion-o lome lusaka lynx-o mamoudzou mandora mbabane monkian montevideo mumm-ra nayda noumea nuuk oranjestad pagopago panthro papeete paramaribo portvila pumyra sainthelier sanaa slithe snowman stanley stockholm tabbuts taipei tashkent thunderkittens torr torshavn turmagar tygra vaticancity vultureman washingtondc willa wizz-ra wollows

MySQL servers:
daggernose homie-vsql48 homie-vsql49 homie-vsql50 homie-vsql51 homie-vsql52 homie-vsql53 homie-vsql54 homie-vsql55 homie-vsql56 homie-vsql57 homie-vsql60 lamnidae lax-maildb3 lax-maildb4 smoothhound speartooth spurdog

Update – 1:15pm: This issue seems to have resolved itself as of about 30 minutes ago. Our admins were checking on our networking and were unable to find any cause for the problems. It doesn’t look like the full list of servers above were affected, just a handful of MySQL machines. If we have any more information as to what caused this we will post it here. At this time though, we don’t know the cause. We’re sorry for any inconvenience this caused you.


3:11 pm

Catcher lost his mitt

Posted (October 24th, 2010 at 3:11 pm PST) by brettg

One of our filers for our spacey cluster is experiencing some booting issues at the moment and our staff is investigating the cause. At this time the following machines may be inaccessible while we get Catcher back online:

grapico hansens hi-c horchata inko izze jolt koolaid lipton minutemaid mrpibb mug musclemilk nestea odwalla oj orangebang pepsi powerade propel ps6937 punch rockstar rootbeer schweppes seltzer slice smoothie snapple sparkletts sprite squirt stewarts sunkist tab tallapolitica tang tea tizer tropicana v8 voss welchs 7up amp aquafina arizona barqs bawls calpico cappuccino caprisun clamato coffee coke dasani drpepper excel fanta frappe fresca gatorade

This will affect mail for some users as well that have users on these machines.

This is only for Spacey Cluster!

If you have any questions please contact our support staff.

Update 18:20 pacific: catcher is back up and things are recovering. The problem was the CF card reader in the head unit of the file server, which required us to replace the head unit. The CF reader was likely dead for a while, but that’s not a problem while the machine is stil running. Looking at the logs it looks like one of the drives failed poorly, took out the fiber channel loop and the machine panic’ed (it panics in this situation to prevent data loss, the nvram still has unwritten data in its memory). We swapped out the head and it’s back up and running just fine!

If you’re still having issues, please contact support!

UPDATE 7:50pm – Unfortunately, there are some network problems on the above server, so mail is still getting queued up until that’s resolved. We’ll update this post again as soon as we make progress on this situation. We apologize for all inconvenience this has been causing today.

UPDATE 9pm PDT – as of about an hour ago, all issues with the machine are resolved. Mail is flowing through again, and logins are working. There may be a bit of delay due to the number of messages that accumulated during the day, but you should see all your incoming mail soon.


5:19 pm

phillip networking trouble

Posted (October 23rd, 2010 at 5:19 pm PST) by jasony

It looks like phillip is having some networking issues after a reboot.  We are investigating this matter and will update this as soon as we have things back up and running.

Update  5:33pm: It seems that the machine booted up under the wrong kernel so after looking through a bit of history and then selecting the correct kernel, phillip is back up and running.  Please contact our support staff if you have any further trouble.


6:00 am

.com/.net registry maintenance 2010-10-23

Posted (October 23rd, 2010 at 6:00 am PST) by kitchen

We’ve received a notification from VeriSign that the .com and .net registries will be down for maintenance Saturday 2010-10-23 from 9:00 to 9:45pm EDT. This means that you will not be able to register or manipulate any .com or .net domain registrations during this period.

Here’s the email we received from VeriSign regarding this maintenance window:

To All Registrars,

The .com/.net Production environment will be taken offline on Saturday, October 23, 2010 EDT (Sunday, October 24, 2010 UTC) for a scheduled maintenance.

Date: Saturday, October 23, 2010 EDT (Sunday, October 24, 2010 UTC)

Time: 2100 hrs – 2145 hrs EDT (0100 hrs – 0145 hrs UTC)

Duration: 45 Minutes

Purpose: Routine Maintenance

We will post an update here once things are working again!


9:18 pm

Large scale networking issues for all DreamHost customers – UPDATE

Posted (October 21st, 2010 at 9:18 pm PST) by Jason C

Some customers may currently be unable to see their sites and have issues accessing core DreamHost services.  This is due to a networking issue that is currently affecting a significant portion of our servers.

Our trusty team of networking ninjas have been dispatched to fix things.  Once we know more about what is going on, we will update this post with all the details we have.

Update on October 21, 2010 @ 10:34 PM, PST: Word back from our admins is that the issue is routing based.  Amongst the assorted problems springing from this is a significant amount of packet loss.  That’s why some folks can see things just fine while others are having really bad problems.  We’re getting really close to cracking this nut — although we still don’t have an ETA — but we thought you’d still like to know what’s going on.

Update on October 21, 2010 @ 10:52 PM, PST: A side effect of the outage has been the violent thrashing of our internal load balancers. Naturally, this made our problems even worse. We’ve leveled those out now and continue to push forward until our networking is back to normal. More updates coming soon!

Update on October 21, 2010 @ 10:59 PM, PST: And we’re back! Apparently, this whole fiasco was caused by a bug in Cisco’s VSS operating system. Sorry that it was so hard for our team to pin down! They’re on standby now tho and will continue to watch the network to make sure nothing goes pear shaped again. If you are still seeing issues, please contact our support team as soon as possible!

Update on October 22, 2010 @ 03:52 AM, PST: Our network went down again for approximately 20 minutes as we had issues with VSS again. We have reverted back and everything is back online as of a few minutes ago.


4:32 pm

Homie-vserver189 having issues

Posted (October 19th, 2010 at 4:32 pm PST) by Leon

Affected VPSes:

ps29163 ps29542 ps33766 ps33767 ps33768 ps33770 ps33771 ps33774 ps33776 ps33778 ps33780 ps33781 ps33788 ps33789 ps33790 ps33791 ps33792 ps33793 ps33794 ps33797 ps33798 ps33800 ps33801 ps33804 ps33805 ps33807 ps33808 ps33810 ps33833 ps33834 ps33838 ps33841 ps33842 ps33843 ps33844 ps33847 ps33848 ps33850 ps33851 ps33853 ps33854 ps33855 ps33856 ps33857 ps33858 ps33859 ps33860 ps34639 ps34773 ps34836 ps34864 ps34869 ps34872 ps34873 ps35008 ps35019 ps35082 ps35099 ps35107 ps35188 ps35221 ps35315 ps35326 ps35363 ps35408

We’re looking into it now. We’ll update shortly when we have a better idea of what is happening to this machine.

UPDATE: All’s well. Turned out to be an attack. We got it covered. If you’re still having issues, please contact your friendly DreamHost support cyborg.


1:52 pm

Shared mysql downtime on homie cluster

Posted (October 19th, 2010 at 1:52 pm PST) by Justin@DH

We are investigating a sudden outage of mysql servers in a specific rack. The admin team are looking into it right now. Possible affected servers:

fatalbert, shaggy, elmerfudd, yogi, spongebob, sardine, tuna, swordfish, cod and also 7 MySQL VPS servers.

— Update Tue Oct 19 14:05:59 PDT 2010 —

The network admins have connectivity restored to this rack, and service is restored. We are watching to make sure everything comes back up also. If you are still having issues related to your database, it may not be related to this issue so please go ahead and contact support.


7:43 am

Shared server ‘biggs’ downtime

Posted (October 19th, 2010 at 7:43 am PST) by patrickm

We are sorry to report that the shared server ‘biggs’ is experiencing some issues which are causing downtime for customers hosted there.  Our Admins have been working hard to resolve the problems there for a few hours now and we expect them to be resolved shortly.  Please check back here for any updates we can provide!

Update 8:14am: We were able to isolate the domain which was the target of a DDoS attack and have taken appropriate action. Most sites will be up shortly, while others may take a few hours for the new IP address to be pushed though cached DNS entries.


 
© 1996-2010, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
10:16 pm

LAX datacenter link down – UPDATE

Posted (October 17th, 2010 at 10:16 pm PST) by JeffH

Due to some unforeseen consequences of our scheduled network maintenance, we have lost our link with our LAX datacenter.  Unfortunately, this means a fairly widespread outage at the moment.  The link actually just came back up a few minutes ago, but since our network engineer is en route to the other location, we can’t say that the restoration of service is permanent yet.  Please be prepared for intermittent downtime as we diagnose and resolve this issue.  We will update this post as we are able.  We apologize for the inconvenience!

Update: The issue was with the core router of our datacenter, and not on our own network.  This has been identified as the problem, and is now resolved.  Again, sorry for the downtime.  Please contact support if you’re having any lingering issues.


6:00 am

homie-vserver09 and homie-vserver29 downtime for Sunday 17th

Posted (October 17th, 2010 at 6:00 am PST) by douglas

Homie-vsever09 and homie-vserver29 will be taken down temporarily at  (6am) Pacific Time, on the Oct 17,  to be moved over to newer, more stable hardware.

The reason for these moves is that homie-vserver29 has a failed drive in it’s RAID array which is refusing to rebuild and homie-vserver09 is using an older RAID card and RAID-6, a very slow combination which has been contributing to low overall system performance, and a rather long fsck not too long ago…

For homie-vserver09, the following will be affected:

ps11946 ps12015 ps12198 ps12512 ps13158 ps13210 ps13233 ps13933 ps14070 ps14071 ps14385 ps14757 ps16765 ps17605 ps17970 ps18342 ps18354 ps18355 ps19682 ps21033 ps21234 ps21623 ps22752 ps4484 ps4492 ps4502 ps4507 ps4510 ps4513 ps4529 ps4537 ps4544 ps4790 ps4807 ps4829 ps4857 ps4927 ps4954 ps5020 ps5698 ps5723 ps6474 ps6527 ps6737 ps6754 ps7422 ps7475 ps7478 ps7479 ps7488 ps7498 ps7947 ps8106

For homie-vserver29, these VPSes:

ps11035 ps11102 ps11126 ps11137 ps11277 ps11323 ps11594 ps11599 ps12547 ps14044 ps14047 ps14049 ps14050 ps14055 ps14107 ps14137 ps14236 ps14279 ps14471 ps14483 ps15211 ps15212 ps15218 ps16414 ps16995 ps18375 ps18445 ps18447 ps18448 ps19297 ps19302 ps20194 ps22433 ps22534 ps22758 ps23213 ps24132 ps5148 ps6821 ps6822 ps6825 ps6827 ps6843 ps6856 ps7006 ps7142 ps7176 ps7377 ps7421 ps7506 ps7840 ps7846 ps7864 ps7867 ps7879 ps8305 ps8710 ps8828 ps9008 ps9038 ps9501 ps9544 ps9776

Update 10-17 7:30am We’re now running the final set of rsyncs with the VPS servers shut off.
Update 10:40am homie-vserver09 is up and working after some fun with ARP. homie-vserver29 still has guests rsyncing, but most have completed— however the raid controller on the old machine went stupid again so we’ll need to restart rsyncs for those that didn’t finish.
Update 12:45pm homie-vserver29 took way longer than it should have due to network gremlins. For now we’re moving the machine back to it’s old hardware till we can resolve the network issue.


11:00 pm

Network Maintenance: October 16th, 17th and 18th – 23:00 – 02:00

Posted (October 16th, 2010 at 11:00 pm PST) by robertn

NOTE:  this maintenance was completed successfully.  We now enjoy Virtualization in our networking core both at LAX and Garland data centers.  This will ensure a more robust network topology.  Thank you for your continued support.

This weekend there will be two network maintenance windows, starting Saturday night at 23:00 until Sunday morning at 02:00 and again on Sunday night at 23:00 and ending Monday at 02:00.  Maintenance  will be on our core routing architecture.

Although no customer down time is expected, there is always a potential for disruption.  Please be forewarned!

If you experience an outage, please contact our friendly support personel.


3:28 pm

Phpmyadmin Disabled Due to Security Vulnerability

Posted (October 15th, 2010 at 3:28 pm PST) by Ralph

Today we had to disable phpmyadmin due to a serious exploit:

http://www.securityfocus.com/bid/34236

We attempted to release a new version but unfortunately it didn’t work and caused errors when use was attempted. For now we are forced to leave phpmyadmin off until we can get the new version working in our system. If you must do database work in this time please check out the following articles on using MySQL at DreamHost:

http://wiki.dreamhost.com/Category:MySQL

Sorry about the inconvenience. As soon as we get this back up again we will update this status post.

UPDATE 10/17/2010 – As of this afternoon, the new version of PhpMyadmin is present on all our servers. Thanks for your patience, and let our support know if you need any help with it.

Update Mon Oct 18 07:09:35 PDT 2010: Please see previous post for further info on the phpmyadmin upgrade and issues.


2:12 pm

MySQL PHPMyAdmin error

Posted (October 15th, 2010 at 2:12 pm PST) by mir

The new patch of PHPMyAdmin is causing this error to appear for customers:

Error

MySQL said: Documentation
#2002 – The server is not responding (or the local MySQL server’s socket is not correctly configured)

We are currently looking into either fixing the issue or rolling back the patch until we can. We’ll update this post as soon as the matter is resolved. Please note that this should not effect other connections to databases (such as sites and command line interface).

Quick update: rather than rolling back the patch we’re working on a proper fix which may take a few hours. Because this is not keeping databases themselves from being accessible via other means (only PHPMyAdmin is effected) it’s best to avoid having many compromised accounts which is what would result from a simple rollback of the patch. We’ll update this post once we have more information.

Update Fri Oct 15 18:14:00 PDT 2010: phpMyAdmin is functional again on all hosting servers.

Update Mon Oct 18 07:09:35 PDT 2010: We are having a few issues with the upgraded phpmyadmin on the frisky and spacey clusters. If your webserver is named like a snack, drink or car, this may affect you. We will update this post when we have further info on a fix for this.

Update Tue Oct 19 12:21:31 PDT 2010: The issue with those last few phpmyadmins were fixed up yesterday and we haven’t see much issue afterwards. We believe this issue is resolved. If you have further issues, please contact support.


3:46 am

mysql and vsql services being worked on

Posted (October 15th, 2010 at 3:46 am PST) by Ken

mysqlbackup decided to snooze again. We are in the process of cleaning up the mess and will update this post when we’re finished.

Update 9:39pm PST: We are in the process of creating new backups for mysqlbackup which is resulting in some intermittent load on the following servers:

anchovy bagheera basking bonito bubsy buck bullwinkle butterfish chapulin cod coelacanth dopey dory dumbo elmerfudd eltigre
escolar fatalbert felix flounder foghorn gill grendel grinch grouper gumby gurgle halibut halosaur inki jetsons leghorn
mackerel magoo midnight mudskipper naroga narwahl neko perch phillip popeye puffer pumbaa salmon sardine sculpin shaggy
sifl slag slowpoke solomon spaceghost spongebob swordfish terrance tetra tib topcat towelie tuna turbo yellowtail yogi zazu
homie-vsql03 homie-vsql08 homie-vsql09 homie-vsql10 homie-vsql11 homie-vsql12 homie-vsql14 homie-vsql15 homie-vsql16
homie-vsql17 homie-vsql18 homie-vsql19 homie-vsql20 homie-vsql21 homie-vsql22 homie-vsql23 homie-vsql24 homie-vsql25
homie-vsql26 homie-vsql27 homie-vsql28 homie-vsql29 homie-vsql31 homie-vsql32 homie-vsql33 homie-vsql34 homie-vsql35
homie-vsql36 homie-vsql38 homie-vsql39 homie-vsql40 homie-vsql41 homie-vsql42 homie-vsql58 homie-vsql59

— Update Sun Oct 17 13:57:23 PDT 2010 —

Unfortunately we are still seeing stability issues with this backup server, despite it being worked on on Friday to upgrade its operating system and check its file system. The admins are still looking into the issue, while our oncall admins are doing what is necessary to try to keep the servers up and available as much as possible. We hope to have a fix within a day or two. If we have more info we will update this post.

— Update Tue Oct 19 12:09:53 PDT 2010 —

The backup server is looking much better after some settings changes, and we didn’t notice any servers having issues last night due to this. We believe this issue to be resolved for now, but of course we are keeping an eye on everything just to be sure. If you are still having issues, it may not be related to this issue. You can contact support with details and we would be happy to take a look.


4:23 pm

Sabretooth FSCK – UPDATE

Posted (October 14th, 2010 at 4:23 pm PST) by brettg

Hi Everybody,

Machine sabretooth booted into a read only file system earlier today and is now undergoing a file system check. The downtime for a normal fsck takes anywhere between 2-4 hours, usually. We will update you as we monitor the machine.

If you have any questions please contact our support staff via the panel.

Update: We apologize for not updating this sooner, Sabretooth is still in the process of fscking as it has a rather large partition to process. We will update you as soon as it has finished.

UPDATE Fri 15 Oct 04:55 PST: Looks like sabretooth has decided to not play nicely. Since it doesn’t want to play with the other guys, we’re replacing it’s hardware. As soon as we have sabretooth back up, we’ll post an update here.

UPDATE Fri 15 Oct 06:36 PST: Due to the raid controller failing during the fsck we moved this machine to new hardware and most services should be restored and all services restored within the next 30 minutes or so. Due to the type of failure we had to restore from backups. This means data uploaded in the last 48 hours will be lost and the server will be slower than usual for about a week until the restore process is complete.

UPDATE Fri 15 Oct 07:03 PST: All services should be restored. Please contact support if you are on sabretooth and unable to access your sites.


5:48 am

Homie-vserver179 being failed over to new hardware

Posted (October 13th, 2010 at 5:48 am PST) by JeffH

Homie-vserver179 went down earlier tonight, and after several attempts to resuscitate it, we’ve determined the motherboard is dead.   The good news is that this very likely won’t be one of those long, painful failovers with restoration from backups, as this machine used our newer, fancier RAID hardware, and thus, we should just be able to swap its drives into a new chassis and have you folks back up and running in a reasonable time period.  Again, sorry for all the hassle tonight, and we’ll do our best to get your back up and running ASAP.

Update: The machine was restored at 8:44am, after updating the ARP tables the VPS machines became visible to the wild internet a little bit later.


3:09 am

MySQL issues on assorted machines

Posted (October 13th, 2010 at 3:09 am PST) by JeffH

A backup machine is having issues at the moment, which is adversely affecting multiple MySQL machines.   Currently on that list are:

grouper felix anchovy halosaur cod jetsons sculpin halibut chapulin foghorn gumby homie-vsql41 homie-vsql50 homie-vsql15 dumbo escolar

(and probably a fair few others named after fish, cats, or famous homie-vsqls)

If you are on any of this servers, MySQL connections may be intermittent while we work out a resolution for the issues with the backup machine.  We apologize for any inconvenience.

Update: The root problem has been resolved, and most of the cleanup is done…however, if you’re still noticing any lingering MySQL issues, let us know, and we’ll be happy to address them as needed.  Again, sorry for the downtime.


3:38 am

MySql server anchovy doing the filesystem check dance – UPDATE

Posted (October 12th, 2010 at 3:38 am PST) by Ken

Anchovy is going through a filesystem check. We regret the inconvenience and will let you know as soon as it’s back up.

UPDATE: anchovy has returned. If you continue to experience problems, please contact technical support via the normal channels.


 
© 1996-2010, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
7:44 pm

homie-vserver29 down for fsck

Posted (October 11th, 2010 at 7:44 pm PST) by jasony

homie-vserver29 is running through a filesystem check at the moment which may take a few hours.  We will post back as soon as things have finished!  If you have any questions, please contact our friendly support staff!

Update 4:14am – This host machine has finished its filesystem check, and guests (VPSs) are restarting as I type this.  Please contact support with any further issues.  Thanks for your patience (and note the  upcoming announcement for this machine on the 17th of this month).


11:00 pm

Network Maintenance: October 10th 23:00 through October 11th 04:00

Posted (October 10th, 2010 at 11:00 pm PST) by robertn

Wrap up: This maintenance went very smoothly.  No downtime for any customers, and our new architecture is almost completed.  Thank you for your support!

We will be doing maintenance in the wee hours of Sunday night/Monday morning this weekend to begin the final stages of our new core architecture transition.  In LAX, we will finalize our cut over to the new cores by moving some mail load balancers and shutting down the old core.

In the downtown data center, we will continue with our transition by moving some security appliances, load balancers, and point to point links to our new architecture.

We do not anticipate any down time to be associated with this maintenance window, as the equipment is already operational.  However, as with all Network Core maintenance,there is potential for an outage.  If you notice any disruption in connectivity, please  contact our Technical Support Team immediately.

Thank you for your continued support as we improve the DreamHost experience!


8:00 pm

Short Downtime DHApps and Easy OneClicks Sunday Night

Posted (October 10th, 2010 at 8:00 pm PST) by kitchen

Sunday night, 2010-10-10 around 10pm pacific we will be taking DHApps sites and all Easy One-clicks down momentarily to upgrade the file server which is hosting the files for these products. We expect a downtime of no longer than 10-15 minutes, but as always, things can go wrong, so we’re putting in for an hour maintenance window.

During this time your sites hosted with DHApps and any and all Easy One-click installs you have will be offline and unavailable. We’ll post here with an update once we’re finished, or if we hit any speed bumps!

Update: 10:00pm pacific: We are starting this now, we’ll post an update once we’re finished!

Update: 10:06pm pacific: We have completed our maintenance and DHApps and Easy One-Click sites should now be back online! If you’re having any trouble with any of these please contact support!


8:12 pm

Server Kigali suffering from a DDOS attack!

Posted (October 9th, 2010 at 8:12 pm PST) by Brian@dreamhost

Unfortunately, our server Kigali is suffering from a DDOS attack. We’re in the process of rerouting traffic, but DNS still needs to be updated correctly. We apologize for this, and hope to have things sorted out ASAP!


6:36 pm

Mailman upgrade @ 6PM PDT tomorrow

Posted (October 6th, 2010 at 6:36 pm PST) by beth

We will be upgrading Mailman tomorrow, October 7th, 2010 at 6:00PM PDT. During this time, all discussion lists will be inaccessible. This process will take approximately two hours.

E-mail and files related to discussion lists (including configurations and archives) will not be lost. All discussion list e-mail will be queued until the upgrade is complete, and delivered at that time.

This will only affect discussion lists. No other services will be affected by the mailman upgrade.

More info:
2.1.14 is a minor bug fix release and you should not need to do anything on your end! For more information regarding this release, see the CHANGELOG.


12:15 pm

Frisky downtime

Posted (October 6th, 2010 at 12:15 pm PST) by andrea

Some of the storage servers for the frisky cluster are currently down and are causing downtime with both web and mail services.

We’re looking into this, and apologize for any inconvenience this is causing. We’ll update this post when there is improvement in the situation.

UPDATE 12:30pm PDT: The affected storage servers are now back up and running. We’re keeping an eye on them to make sure all issues were resolved, but you should already see improvement in your web and mail services.

UPDATE 1:50PM PDT: We can now confirm that the storage servers are functioning correctly. We’re in the process of checking all web servers connecting to them to ensure that all sites are loading fine. We appreciate your patience!

UPDATE 6pm PDT: All affected web servers have been up and running for a couple of hours, so we’re closing this post. If you have any questions or need assistance, please contact support through the panel under Support/Contact Support.


2:42 am

MySQL server ‘solomon’ down

Posted (October 6th, 2010 at 2:42 am PST) by patrickm

We are experiencing an outage with the MySQL server ‘solomon’.  Our team is working to resolve this downtime now and we will post here with updates when they are available.

Update 4:06am – This MySQL server is back up now!  The issues that caused the downtime were due to the network configuration on the server and then a bad network cable was found.  Both issues resolved, so please contact Support if you are still having any issues making a connection to this server.


3:13 pm

Server Carvin – Moving to New Hardware

Posted (October 3rd, 2010 at 3:13 pm PST) by brettg

Carvin today decided to get a little sick and we’ve had to start failing the machine over to a new set of hardware.

We’re currently in the process of moving the machine and we will update you with more information as we get the new machine setup.

Update: We’re having to restore this machine’s /home data from backups because the /home partitions had degraded data. Your users should be up and running right now, while we move the data back into place and your websites should be online as we mount your users directory to our backup data. If you have any questions or are experiencing issues unrelated to this specific outage, please contact our support staff.


3:00 pm

Network Maintenance Weekend! October 1 through October 4

Posted (October 1st, 2010 at 3:00 pm PST) by robertn

The retiring of our old core is not complete, as there are a number of legacy systems and stand alone appliances (like our mail load balancers) that cannot be smoothly transitioned at this time.   This did cause a mail outage and some older servers to be unavailable for a time.  We apologize for any inconvenience this has caused.

We are retiring our old core tonight, implementing a new border in LAX in preparation for additional transit to be installed in the coming months.  This should be transparent to our end users.  However, as with all network architectural changes, there is a possibility for intermittent outages.  Forewarned is fore armed…

================================================================================================

On Sunday, we will be transitioning our Garland data center to our new VSS core technology.  As with the LAX transition, this is planned to be non-intrusive to our users.   However, as with all network architectural changes, there is a possibility for intermittent outages.  Forewarned is fore armed…where have I heard that before.

As always, if you experience an extended outage, please let us know immediately.


4:40 pm

DreamHost PS/VPS Stability Issues

Posted (September 27th, 2010 at 4:40 pm PST) by Ralph

We are currently experiencing problems with our VPS service. After upgrading the kernel on all of our VPS host machines we have been having problems with them staying stable and functioning properly. We are currently working on patching previous kernels that we noticed better stability with. We hope to have a new kernel ready and deployed tomorrow to help limit the stability issues many of you are experiencing. We will update this post with further information as we have it.

Update Sept. 28 09:00 PST: We have backported the security fixes to the kernel revision which was noticeably more stable and initial testing is looking positive.  We are building system profiles and comparing them to the current kernel series to make sure we nailed the problem.  Once we are confident in the new kernel we will begin deploying it across our VPS clusters.


 
© 1996-2010, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
6:36 pm

Domain registration services are down at the moment due to planned maintenance by verisign

Posted (September 25th, 2010 at 6:36 pm PST) by jasony

Woo! Domain registration services are down at the moment due to planned maintenance:

“To All Registrars,

The .com/.net Production Maintenance will begin as scheduled at 2100 hours EDT. We anticipate this maintenance period to last forty five (45) minutes.

If you require further assistance, please contact us at info@verisign-grs.com.

Best Regards,

Prakash Karappurath
Customer Service
VeriSign, Inc.
www.verisign.com
info@verisign-grs.com

If you have any questions, please contact our friendly support staff!

Update Sep 25th 6:55pm PST: Looks like its all done and everything looks to be working normally.  If you have any issues, please contact our support staff!


9:34 pm

MySQL Kernel changes (part 2!)

Posted (September 24th, 2010 at 9:34 pm PST) by jasony

Over the next week or two, we will be changing the kernels on a few of our MySQL servers.  The kernel change will require a reboot once the changes have been applied which will cause, roughly, a 10 minute downtime window for your MySQL services. These changes will begin Friday, Sept 24th  after 8:00 PM PDT and will be done in alphabetical order with the server names posted below.

chimaera hammerhead knifetooth mako megalodon nemo nursehound porbeagle silvertip swellshark wobbegong wodnika

If you have any questions, please contact our support staff!

Update – Sept 24th 9:14pm PST: chimaera is finished (this will be the only server for tonight), please contact our support staff if you are encountering any issues at all!

Update – Sept 27th 8:55pm PST: hammerhead, knifetooth, mako, megalodon are finished

Update – Sept 28th 7:53pm PST: nemo, nursehound, porbeagle, silvertip, swellshark, wobbegong, wodnika are finished

That’s it!  All done!   Wooo!

If you are having any issues or have any questions, please contact our support staff!


7:30 pm

Vservers in fsck

Posted (September 22nd, 2010 at 7:30 pm PST) by Leon

The following vservers are affected:

ps14197 ps14204 ps14895 ps15197 ps15198 ps15199 ps15200 ps15203 ps15204 ps15809 ps26814 ps26866 ps27090 ps27093 ps4743 ps4748 ps4752 ps4755 ps4762 ps4764 ps4775 ps4778 ps4794 ps4803 ps4824 ps4833 ps4843 ps4851 ps4860 ps4874 ps5006 ps5012 ps5061 ps5079 ps5084 ps5087 ps5107 ps5110 ps6808 ps6817 ps7727 ps7729 ps7869 ps8193 ps8231 ps8249 ps8429 ps8433 ps8442

This should be resolved within an hour or so.

Update: The Fsck has finished, if you’re noticing further problems, please contact our support staff.


3:00 pm

Panel issues

Posted (September 22nd, 2010 at 3:00 pm PST) by mir

We are aware of the problems effecting the account control panel and our admin team has been working to bring it back online. We’ll update this post as soon as we have more detailed information.

– UPDATE –

Two of our panel machines went down simultaneously, causing a bit of overload for the rest of the pool. These machines have been fixed and are back up serving the panel. If you continue to experience problems or slowness with the panel, please notify support.


6:39 am

Spacey file server downtime

Posted (September 22nd, 2010 at 6:39 am PST) by Justin@DH

One of the many file servers for the space cluster has stopped responding. Our admins are on their way to check out what is going on with it and to get it back up and running. This affects some but not all customers on the following webservers:

7up aef amp aquafina arizona bawls boba brisk cactuscooler cappuccino caprisun chai clamato codered crush dads dasani dietrite drpepper eggnog espresso evian fiji frappe fresca fuze gatorade geyser grapico hansens hi-c hiball horchata ibc inko izze koolaid lipton longcat martinellis minutemaid monster mountaindew mrpibb musclemilk nastia nehi nesbitt nestea niagara odwalla oj orangebang ovaltine painame pepsi perrier propel punch redbull refresco riptide rockstar rootbeer schweppes seltzer servidorgva shasta silk slice sly smoothie snapple sobe stewarts sunkist sunnyd tab tampico tang tea tizer tonic tropicana v8 voss welchs yoohoo

— UPDATE —

The file server is back up, and service is restored. If you have any remaining problems related to the downtime, please contact support.


4:48 am

Austin Has Amnesia

Posted (September 21st, 2010 at 4:48 am PST) by JeffH

Austin went down with a hardware error earlier, and it looks like one of the memory sticks may have gone bad.  We’re testing to determine which one so we can replace it.  We’ll have it back up as soon as possible.  We do apologize for any inconvenience.

Update 5:46am PST - Ah, I just remembered!   Bad memory has been replaced with thoughts candy and double rainbows.  Austin is back up and running.  Sorry for the downtime.


7:07 pm

Guatemalacity Being Failed Over to New Hardware

Posted (September 20th, 2010 at 7:07 pm PST) by Leon

We’re looking into what’s causing guatemalacity to be down right now, looks like an issue with it’s networking config. We will update when it’s coming back up.

Update – 10:07pm PST: Unfortunately it looks like we are going to have to move guatemalacity to new hardware. We will update once we have things moving along.

Update – 12:58am PST: The drives from the old machine have been transferred over, and the new machine is being configured to takeover guatemalacity’s identity.  The process itself takes a bit longer than getting a machine to the point of restoring backups, but it’s worth it.   The payoff is that for a few hours downtime, you avoid a few days of poor performance and high machine load while backups transfer over the network.   Pity those poor folks over on budapest, eh?  Thanks for your patience!

Update – 1:50am PST: Told ya it would be worth it, didn’t I? The server is back up, and sites are loading again.  If you’re having any lingering issues, please submit a ticket and let us know!


4:06 pm

Budapest down for fsck

Posted (September 20th, 2010 at 4:06 pm PST) by Leon

Unfortunately with all of today’s craziness, we’ve seen that Budapest did not quite make it. It’s currently undergoing an fsck which could take anywhere from an hour to a few hours. We’ll update when it’s back up.

Update  – 08:43pm PST: It looks like the fsck failed and currently going to move the machine to new hardware.  Will post back once that is completed.

Update – 11:45Pm PST: The server is currently being restored via backups and could take a few days for this to complete.  We will update once the restore has finished.  If you have any questions, please contact support.

Update – Sept. 21st 14:00 PST: The server is showing some high loads and had unresponsive apache servers earlier. We were having issues with mounting but those have been resolved (which allowed us to fix the apache issues). The load will continue to fluctuate as we restore via backups. We’ll update this post again when we have additional information.

Update – Sept. 23st 23:10 PST: All users data has been restored and the backup server is no longer being mounted. Load average and I/O access should now be normal speed and there should not be any excessive slowness anymore.


1:38 pm

Site downtime on various servers

Posted (September 20th, 2010 at 1:38 pm PST) by danwy

We’re currently investigating load issues on some of our servers that is causing downtime for customer sites. If your site is down at the moment, this is likely the cause. Unfortunately, we don’t have many details at the moment on what servers are affected, but our admin team is currently investigating the issue and hope to have more information soon. We will update this post as soon as we have any more information. We apologize for the inconvenience this causes you.

Update – 3pm (PST): This issue looks like it has been resolved. Unfortunately, we’re not quite sure what the issue was. There were no networking issues and nothing new has been pushed out. We did notice that the servers with the highest loads were in one specific datacenter, but those loads began dropping as soon as we started investigating things. We’ve tested dozens of affected domains across several servers and have verified that things are working again. If you continue to have issues, please contact support to let them know. As it is likely not related to this problem.


2:15 pm

Incoming mail delay on junk filter machines

Posted (September 17th, 2010 at 2:15 pm PST) by brians

Mail that’s relayed through our junk filter machines are being queued for delivery. As a result, incoming messages are being delayed. Again, the outage only pertains to users who have enabled our junk filter. You can verify whether the filter is enabled by visiting the Mail::Anti-Spam section of our Web Panel https://panel.dreamhost.com/index.cgi?tree=mail.junk&. Our admins are currently investigating this matter. We sincerely apologize for any inconvenience this has caused and appreciate your continued patience. Please check back for future updates.

UPDATE:This issue was resolved shortly after the initial post. There was a misconfiguration in the software that we use to scan messages. There wasn’t any mail loss, messages were just delayed. Mail is now being instantly processed on the filter machines. Again, we sincerely apologize for any inconvenience the mail delays may have caused. If you are still noticing problems, please contact support.


 
© 1996-2010, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
2:12 pm

PS Reboots

Posted (September 15th, 2010 at 2:12 pm PST) by Ralph

We will be rolling out PS server reboots today in order to update our service. Once the reboots and updates are complete we will post here again with further information on why we performed these reboots. This affects all PS’s.

Update 5:49 pacific: We finished these reboots at around 4:30. There was a privilege escalation vulnerability that was released today that affected the vserver kernels. This was a very serious vulnerability so we had to make the change and reboot the machines immediately with little to no information/warning. Sorry about that. Good news is that you are now safe from the privilege escalation vulnerability. If you still have problems please let our support staff know.


7:48 pm

Our web panel is down …

Posted (September 13th, 2010 at 7:48 pm PST) by Brian@dreamhost

We’re having a bit of trouble with our web panel administration area, currently. We’re working on the problem, and will update as soon as we get more info about a fix! Sorry about the problems!

Update: We’ve restored services to our panel. We apologize for the inconvenience.


9:51 am

luanda filesystem check fun!

Posted (September 8th, 2010 at 9:51 am PST) by douglas

Luanda was rebooted earlier this today and the fielsystem came back read-only. We did a regular filesystem check and all was happy, until we rebooted it and the filesystem started throwing up errors again. So now we’re doing a thorough filesystem check to sort this whole mess out. All the user data is intact so no need to worry there (and of course, there is the nightly backups too).

This filesystem check will likely take a few hours to complete, unfortunately. We will update when it is done.

If you feel this may be affecting you, please check your web panel to figure out what server you are on (panel > Account Status > Your Web Server). For any other servers, please create a support ticket.

Update 2:30pm Fsck finished “Pass 1″ several hours ago. It’s currently working on “Pass 1B: Rescanning for multiply-claimed blocks” and has been at that for 2 hours and 10 minutes now. The bones suggest 3-5 more hours are likely.

Update: This has finished, please contact our support staff if you have any further issues.


9:36 am

Stringray unresponsive — now its back with it’s sting

Posted (September 7th, 2010 at 9:36 am PST) by mir

We’ve dispatched one of our admins to the physical location of the server to work on getting it back up and we’ll update this post as soon as we have a handle on the situation and service is restored.

Update:The system suffered a bad power supply. We have replaced the system with a new machine and it is working normally. There are still a number of configs to be run so there may be a slight system slowdown, but other than that everything is working normally. Stingray is a smaller shared web server.


6:28 am

Webserver ‘constantine’ under DDoS attack

Posted (September 3rd, 2010 at 6:28 am PST) by Steven

Due to the attack sites were not responding.

Update: This is now resolved and sites are accessible. If you continue to see any trouble with your sites, though, please contact support for further assistance. Thank you for your patience!


7:24 am

Server vientiane unresponsive

Posted (September 2nd, 2010 at 7:24 am PST) by mir

We are currently dealing with a hardware issue on the “vientiane” server. We are replacing some of the hardware and expect to have service restored in the next 40 minutes. Thank you for your patience!

Update 08:07 PDT: Ok, this has been moved to a new set of faster/better hardware and everything is back to normal. Please contact support if you are still having any problems and are on the web-server vientiane.


4:41 am

Webserver: ‘Kustom’ hardware failure

Posted (September 2nd, 2010 at 4:41 am PST) by Steven

We are currently dealing with a hardware issue on the “kustom” server. We are replacing some of the hardware and expect to have service restored in the next 40 minutes. Thank you for your patience!

Update: 05:16 AM PDT: The server has been moved to shiny new hardware and is back to serving pages normally again. Please contact support if you are still having problems with your websites and you are hosted off ‘kustom’


11:00 pm

LAX Network Maintenance — today at 11 pm

Posted (August 31st, 2010 at 11:00 pm PST) by robertn

Further Note: Things took a turn for the worse after 4 am PDT today.  Our apologies for the inconvenience!

The new border became unstable due to high cpu load.  This caused routing issues that disrupted service.     We have reduced the load on the interim border router at LAX, and normalize our network.

============================================================================================================

NOTE: This task was completed successfully. Nynjoy your shiny new bandwidth!  RKN @ 3:45 AM 9/1/2010

We will be installing a new border router and transit bandwidth to LAX.  All those hosted in LAX will see an improvement in throughput.

There should be no impact on your DreamHost experience while this occurs.  However, as with all network maintenance, there is potential for outages.  Plan accordingly.


11:00 pm

Network Maintenance LAX

Posted (August 29th, 2010 at 11:00 pm PST) by robertn

We will be rolling out two shiny new cores in our LAX data center.  While we don’t anticipate down time during this transition, best to plan for the worst and hope for the best.

That being said, there may be intermittent outages from 11 pm until 2 am tomorrow.

Update:  Our fantastic networking guy is still tweaking things a bit, but the worst of it should be over.  Might be a few minor hiccups, but they should be short-lived.  Thanks for your patience.  Shiny new cores are shiny.


11:15 am

berlin syn flood — resolved

Posted (August 22nd, 2010 at 11:15 am PST) by douglas

berlin is getting sync flooded at the moment which are affecting several sites on the server. The sync flood is not negatively impacting our network as a whole, just the single machine. We are working to resolve this.

Update: The problem has been isolated and has been forwarded off to our abuse department. DDoS SYN flood is now over on berlin. Let the air raid commence!


 
© 1996-2010, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
1:41 am

Muscat Hardware Failure

Posted (August 17th, 2010 at 1:41 am PST) by JeffH

Muscat appears to have experienced a hardware failure, and is currently unresponsive.   A tech is currently looking into the issue, and the hardware will be failed over if necessary.   We apologize for any downtime this causes, and will work to bring a swift resolution.   We will update with details as they become available.   If you are having issues and are not hosted on Muscat, please contact technical support.

Update 3:55am PST: Failover is nearly complete, and no data was at risk.  Will update when machine is back online, but things are looking good.  Sorry for the downtime, but we’ll get you back up ASAP.  Thanks for your patience :-)

Update 5:57am PST: Muscat is tentatively back up and running.  It still seems to be suffering some instability, but we’d rather have it running in this state while we work on a more permanent solution than just leave everyone’s sites down.  We will update this post as the situation warrants, but Muscat is up and serving sites again.   Thanks for your patience.

Update 6:21am PST:  It appears the new hardware is failing as well, which means we’ll need to do a failover from backups.  This will take longer (which is why we try to avoid it in the first place), but it is a permanent fix, as none of the old hardware will be used.  We do apologize for the hassle, and will have things back up and running as soon as we possibly can.

Update 7:19am PST: This server is now back up and running on the new hardware! If you are still having any issues with your hosting, please contact Support for assistance!


8:40 am

homie-vserver09 issues — RESOLVED

Posted (August 16th, 2010 at 8:40 am PST) by douglas

homie-vserver09 went down today due to some odd behavior which forced us to powercycle the machine.

Currently the system is fscking which can take several hours, depending on how full the disks are and what kind of issues are found by fsck.

We are also preparing a fail-over machine for homie-vserver09 as well. Even if the current old machine comes up without a hitch, we’ll still be moving it over to new hardware due to concerns over the RAID controller.

This affect a fraction of a percent of our PS customers, so the chances are if you’re just casually browsing this post, it doesn’t affect you, unless, of course, your ps is listed below:

ps11946 ps13233 ps16765 ps19682 ps4492 ps4537 ps4857 ps5698 ps7422 ps7947
ps12015 ps13933 ps17605 ps21033 ps4502 ps4544 ps4927 ps5723 ps7475 ps8106
ps12198 ps14070 ps17970 ps21234 ps4507 ps4790 ps4954 ps6474 ps7478
ps12512 ps14071 ps18342 ps21623 ps4510 ps4807 ps5020 ps6527 ps7479
ps13158 ps14385 ps18354 ps22752 ps4513 ps4829 ps5122 ps6737 ps7488
ps13210 ps14757 ps18355 ps4484 ps4529 ps4836 ps5630 ps6754 ps7498

Update: File system check was completed in roughly 4 hours. We will still be moving this machine over to newer hardware with a faster RAID card in a week or so after the first few rounds of rsyncs finish. During the hardware move there will be minimal downtime, though that’s for another post :)


4:10 pm

Noumea having hardware problems

Posted (August 15th, 2010 at 4:10 pm PST) by andrea

Our admins are looking into fixing noumea, sorry about the inconvenience this downtime is causing. We’ll post here again as soon as there is progress.

Update: This should be all cleared up by now, if you notice further problems, please contact our support staff via the panel.

UPDATE 6:40pm PDT: The machine is now fixed and restored from backups. Unfortunately, anything uploaded since yesterday evening when our last backup ran, will have to be re-uploaded. We apologize for the inconvenience.


12:23 pm

MySQL Server “tiny” downtime

Posted (August 15th, 2010 at 12:23 pm PST) by Justin@DH

The mysql server tiny stopped responding. A remote reboot didn’t go through and an admin had to manually reboot it at the datacenter. We’ll be checking out the server hardware to make sure it is ok or replace it in the near future if not.

Tiny is back up, if you have further issues with it, please contact support.


11:12 am

ryoohki, pichu, nukunuku – Blingy mysql downtime RESOLVED

Posted (August 15th, 2010 at 11:12 am PST) by Justin@DH

These 3 older mysql servers are currently down, our admin team is working to get them back up. When we have further info and they are back up, we will update the post. Sorry for the downtime.

Servers are back up.


11:11 am

Maputo is Unavailable

Posted (August 13th, 2010 at 11:11 am PST) by Terri

The shared server, maputo, just had a public NIC failure, which will cause it to be unavailable. We are sending a technician to replace the NIC. We estimate that the machine will be back up and running in about an hour.

This does not affect email, WebPanel, or any other service other than website access for customers residing on server maputo.

We apologize for the inconvenience. We will keep this space updated.

Update: maputo is back up and running. We’re keeping an eye on it to make sure the NIC is working perfectly.


7:40 pm

Hagstrom Broke a String

Posted (August 12th, 2010 at 7:40 pm PST) by Leon

The hagstrom web server had its hardware fail and we moved it over to new hardware. We apologize for the status post after the incident but everything should now be back up without any data loss. If you have any questions, please contact support.


5:39 pm

Ferrari Crashed

Posted (August 10th, 2010 at 5:39 pm PST) by Leon

We’re currently experiencing hardware issues with the ferrari web server. It’s currently being moved to new hardware and we’ll update everyone once it’s back up!

Update 6:56PM PDT: Ferrari is now back in action with its new hardware. All services should be restored but if you’re still experiencing issues, please contact support!


8:41 pm

Unscheduled filesystem check on eskimos — resolved

Posted (August 9th, 2010 at 8:41 pm PST) by Jason C

Users on eskimos are probably seeing issues with their sites at the moment. This is because our admins had to start an unscheduled fsck of the server.  Once the system finishes this check — in roughly 30 to 90 minutes — all sites on the server should recover.

We’ll let you know what’s going on as things progress!

Update – Aug 10th 5am: Filesystem check found inconsistencies and is repairing them, so while this will not take days and days and days, it may be a few more hours.  We apologize for the inconvenience, and will get you back up and running as soon as possible.

Update – Aug 10th 12pm: Unfortunately, this is taking longer than expected to complete. There’s not much else we can do until the fsck is complete so that data doesn’t get corrupted. We will update this post as soon as it is finished.

Update 3:00pm: Hurray! The server is back up! We will be looking a scapegoat and sacrifice him to the internet gods. Seriously though, we will be investigating this matter.


5:32 pm

Mail problems for homiemail-master

Posted (August 3rd, 2010 at 5:32 pm PST) by Ralph

We unfortunately had a problem where some users were removed from our mail database for homiemail-master. If you are unsure what mail server you are on you can check the “account status” tab on the DreamHost Web Panel.

We are investigating why users were removed from the database. It appears we had a slight problem with accounts appearing inactive even though they were not. Our developers are working to fix this issue permanently.

All users should be added back into the system within an hour.


 
© 1996-2010, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
1:15 pm

homie-vserver116 down for fsck – resolved

Posted (August 2nd, 2010 at 1:15 pm PST) by douglas

homie-vserver116 filesystem has been reporting errors, and so the operating system didn’t want to mount the system with write enabled. All of your files should be fine and happy. Just a few things weren’t cleaned up on the disk when the plug was pulled.

Update: fsck has been completed and the system will be back online in a few minutes.

Affected guests are:

ps17012 ps17037 ps17111 ps17122 ps17134 ps17191 ps17249 ps17287
ps17322 ps17341 ps17377 ps17413 ps17445 ps17472 ps17483 ps17540
ps17580 ps17612 ps17709 ps17728 ps17752 ps17770 ps17863 ps17870
ps17871 ps17882 ps18236 ps18255 ps18256 ps18284 ps18303 ps18503
ps18504 ps18506 ps18783 ps18794 ps20120 ps23013 ps24268 ps24488
ps25123 ps25652 ps25730 ps25765 ps25789 ps25878 ps25948 ps26065
ps26071 ps26163 ps26164 ps26168 ps26184 ps26450 ps26823 ps17000


11:38 am

Rails 2.3.8 is coming!

Posted (August 2nd, 2010 at 11:38 am PST) by Terri

We’re going to start putting together and testing out the new Rails 2.3.8. So, as usual, you should all freeze your rails!

Freezing is easy! You need to ssh into your account, cd over to project, then run: rake rails:freeze:gems – and that’s it!


8:31 am

Panel updates are being delayed

Posted (July 30th, 2010 at 8:31 am PST) by brians

All changes ( ex. domains additions, password updates, etc ) made through our Panel http://panel.dreamhost.com are being delayed. This is a result of the ” automatic ” WordPress upgrades being performed on customers domains ( Our previous status post ). Unfortunately , I cannot provide an exact ETA. You will likely these see delays for the next couple of hours. Please bear with us. Our developers are also devising a solution that would prevent similar occurrences in the future. We sincerely apologize for any inconvenience this has caused. Your continued patience is really appreciated. Please check back for future updates.

UPDATE: This issue was resolved days ago. Sorry for the delay in updating the post. If you are still having problems, please submit a support request.


5:07 am

WordPress 3.0.1 OneClick Install Auto Upgrades Delay

Posted (July 30th, 2010 at 5:07 am PST) by patrickm

WordPress has released version 3.0.1 to address some minor fixes.  We have updated the OneClick Install for WordPress and you can upgrade your installs now.

Those who have the automatic upgrade option selected will receive this upgrade, but the process is delayed due to the sheer volume of upgrades being processed.  The process is not stalled, just taking some time to complete.  Please be patient and you will receive an email notification when the upgrade has been completed!


12:59 pm

Private Server Upgrades

Posted (July 29th, 2010 at 12:59 pm PST) by Kyle

Jul 29, 2010 @ 12:59:

We are going to begin a wholesale upgrade of private servers in our homie cluster this coming Monday, August 2nd.  The primary motivation for these upgrades is to alleviate the burden and latency associated with backporting security fixes to the no longer supported Debian “Etch” release.

A nice plus is that all the latest and greatest software supported by Debian “Lenny” is now within your grasp.  We are initially doing this only on managed private servers, meaning PS’s that do not have an Admin User setup here <https://panel.dreamhost.com/index.cgi?tree=vserver.adminusers> or SSH Keys setup here <https://panel.dreamhost.com/index.cgi?tree=vserver.keys> and here <https://panel.dreamhost.com/index.cgi?tree=vserver.settings>.

Additional Information:

  • Initial runs of our upgrade script are averaging at about 15 minutes from start to finish.
  • PHP4 has been phased out, it’s been unsupported since 2006.
  • The old copy of your private server’s files will be under /old in case you still need something, we’ll eventually prune these but it’ll be announced here.
  • Customers with unmanaged private servers may request that we preform an upgrade on their machines through support who will relay the request to my queue.

12:35 am

Power strip maintenance 07/27 *UPDATE 2* RESOLVED

Posted (July 27th, 2010 at 12:35 am PST) by Ken

The following servers will be taken down to upgrade the power strips on Tuesday July 27 around midnight.

bcrich, charvel, cort, danelectro, dean, dobro, epiphone, ernieball, esp, fender, fernandez, godin, gretsch, guild, hangstrom, ibanez, kalamazoo, jackson

Maintenance will take approximately 2-3 hours. We thank you for your patience while we strive to improve our services for you, our valued customers.

UPDATE 28 July 02:05 PST: Beginning maintenance. Servers will experience a short outage while power strips are updated. Thank you for your patience while we work to improve your DreamHost service.

UPDATE 2 28 July 04:08 PST: Maintenance is complete. If you are still experiencing issues with your server, please contact Technical Support via the normal channels. Thank you for your patience and have a Happy DreamHost experience.


12:23 pm

Django 1.2.1 (finally)

Posted (July 26th, 2010 at 12:23 pm PST) by AndrewF

Django 1.2.1 packages have (finally) been built, and will become available on most web servers within the next 24 to 48 hours.

Details on the contents of this update, which includes several non-backward-compatible changes, are available from the Django Project web site at:

http://docs.djangoproject.com/en/dev/releases/1.2/

If you run a Django application on DreamHost’s servers, please read and acquaint yourself with these changes, as they may affect you. If you do not run any Django applications, this update will not affect you in any way whatsoever.


5:25 am

Powerstrip Maintenance 7/24

Posted (July 23rd, 2010 at 5:25 am PST) by Sandon

We will be taking the following dedicated servers down while we do maintenance to a power strip which should take about 30 minutes. This will effect the following machines:

auction
bean
bmw
bscmanagement
cowboy
dante
gnu
killeroldies
lifetimesvoice
lowest-airfare
meep
motmot
padrino
saints
tekhost

We will take the machines down between 1 and 2 AM PST on July 24th.

Update 2:30 AM PST: Sorry for the delay. We are taking the machines down now to start the maintenance.

Update 3:25 AM PST: The maintenance took a little bit longer than expected but all the above machines are back up and running. If you are still having problems and one of the above servers is yours please contact support.


11:15 pm

LAX Network center experiencing some intermittent issues **UPDATE**

Posted (July 21st, 2010 at 11:15 pm PST) by Ken

It looks like our network center at LAX is having some minor issues with packet loss. Our ueber-admins are currently looking into the issue. We will update this status post as more information is available.

Thank you for your patience while we resolve this issue.

Update 1:05 AM PST: It appears that the problem was caused by a DDoS attack that was not bandwidth intensive (hard to find) but was causing high CPU usage on our LAX core router. The attack has been blocked and services should be restored.


11:13 am

Mailman upgrade @ 6PM PDT today

Posted (July 20th, 2010 at 11:13 am PST) by beth

We will be upgrading Mailman today at 6:00PM PDT. During this time, all discussion lists will be inaccessible. This process will take approximately two hours.

E-mail and files related to discussion lists (including configurations and archives) will not be lost. All discussion list e-mail will be queued until the upgrade is complete, and delivered at that time.

This will only affect discussion lists. No other services will be affected by the mailman upgrade.

More info:
2.1.13 is a minor bug fix release and you should not need to do anything on your end! For more information regarding this release, see the CHANGELOG.


 
© 1996-2010, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
11:22 pm

vientiane has lost its will to live

Posted (July 18th, 2010 at 11:22 pm PST) by Ken

Our admins are currently working to move vientiane to new hardware. We appreciate your patience and will update this post as soon as we have more information.

Update: The server is back up and kicking, after moving the crucial bits over to another machine, and updating some configuration goodies.  Very sorry for the downtime, but things went smoother than they could have.  The panel issues were not related, but should be resolved as well.  If you’re having further issues, please contact support.


2:21 am

vilnius is losing it’s mind *UPDATE*

Posted (July 12th, 2010 at 2:21 am PST) by Ken

vilinus decided to forget one of it’s drives. We are currently working on the issue and will update this post as soon as we have more information. Please bear with us while we work out this problem.

UPDATE 03:51 July 12 PST: Our resident raid psychologist has had a session with vilnius, and vilinus has now returned to duty. Thank you for your patience while we work these out.

If you still experience issues with your domains, please contact technical support through normal channels.


6:56 pm

Belmopan Undergoing a Filesystem Check *UPDATE*

Posted (July 8th, 2010 at 6:56 pm PST) by Leon

Just like the previous server, wario, belmopan is undergoing a filesystem check as well. Again, depending on how much data is located on the drive, the check can take up to a few hours. We’ll update everyone once it’s done but if you have any further questions, please contact support.

UPDATE July 09 02:38 PST: FSCK has finished and belmopan has returned to life. We will be monitoring to make sure it stays up. If you continue to experience problems, please contact technical support via the usual channels. Thank you for your patience while we strive to improve your Dreamhost experience.

UPDATE July 09 08:28 PST: The server mounted read-only again so we’ve decided that the best solution is to move it to entirely new hardware and fix this one and for all. We are preparing the new server now and should start this process in the next couple of hours. This post will be updated when new information is available.

UPDATE July 10 07:28 AM PST: This server has been back up and running correctly around an hour after the previous status post was made but a status update was not made which I apologize for. Data is currently being copied over and should take about a week from the 9th to finish.

UPDATE July 13 9:21 AM PST We are still encountering intermittent load spikes on the machine. This is the result of the RAID rebuild and data moves running. The RAID rebuild is at 34% and should finish within the next 24-36H. The data restore should be completed within 72H. Again, we sincerely apologize for any inconvenience this has caused and appreciate your continued patience.

UPDATE July 14 12:31 PM PST The RAID rebuild is at 91%. it should be completed shortly. Please continue to bear with us. Thanks again for your continued patience.

UPDATE July 16 09:45 PM PST My apologies for the lack of update, the rebuild was completed yesterday so service should have already returned to normal, thank you for your patience with this issue!


6:34 pm

Web Panel experiencing problems

Posted (July 8th, 2010 at 6:34 pm PST) by Ralph

Our web panel is currently experiencing problems likely due to some development work we did on it recently. We are currently investigating the issue and will update here with any new information we find. Sorry about the inconvenience.


4:39 pm

wario down for file system integrity check *UPDATE 3*

Posted (July 8th, 2010 at 4:39 pm PST) by douglas

Wario has been complaining loudly about file system errors. We’ve switched the system over to single user mode and are now running file system checks. As soon as that is completed, we’ll restore access. Depending on how full the disks are and what needs to be fixed, this can take a few hours.

Update: The fsck finished awhile ago, and we’ve been monitoring the server since then. Everything appears to be good, if you notice further problems, please contact our support staff!

UPDATE Friday Jul 09 00:45 PST: The raid is still having issues. We are currently moving wario to new hardware since it appears the raid card is going bad. Please bear with us while we resolve this issue.

UPDATE Friday Jul 09 03:43 PST: Most services should be restored on the server. The user data is now being mounted from the backup server which is about a day old. The server will run at reduced performance while user data is being restored back to the new raid array which typically takes approximately 1 week but all sites/services should otherwise behave normally (other than performance).

If you are on wario and are still having problems with your site please contact support.

UPDATE Friday Jul 16 12:40 PST: Thank you for your patience, the restore is completed and performance should be back to normal!


12:28 am

Penguin is being moved to new hardware

Posted (July 8th, 2010 at 12:28 am PST) by Sandon

The web-server ‘penguin’ is being moved to new hardware due to its 3ware raid controller refusing to rebuild the array leaving it in a working but ‘degraded’ state. We are moving it to a new machine with a different brand raid controller so it can be back in its normal state that allows for redundancy and disk failures. Just about all the data has already been copied to new hardware but we will need to take the machine down for about an hour to do the final sync of the data.

ETA of downtime is approximately 1.5 hours or less and should start in the next 40-60 minutes. This status post will be updated when the machine is brought down and then back up.

Update 01:32 AM PST: Penguin was just brought down. It should be brought back up within the hour and another 30 minutes or so to get things configured on the new OS/install and hardware.

Update 02:48 AM PST: The final sync took a little longer than expected on this machine as well (by about 10 minutes). The machine is back up and running on the new hardware but still needs to be configured. Services should be restored within 30 minutes or so. Another update will be made when all services are restored.

Update 03:13 AM PST: Ok, all services should be restored on ‘penguin’ if you are on penguin and still having issues please contact support.


12:27 am

Spiderman being moved to new hardware

Posted (July 8th, 2010 at 12:27 am PST) by Sandon

The web-server ‘spiderman’ is being moved to new hardware due to its 3ware raid controller refusing to rebuild the array leaving it in a working but ‘degraded’ state. We are moving it to a new machine with a different brand raid controller so it can be back in its normal state that allows for redundancy and disk failures. Just about all the data has already been copied to new hardware but we will need to take the machine down for about an hour to do the final sync of the data.

ETA of downtime is approximately 1.5 hours or less and should start in the next 30-40 minutes. This status post will be updated when the machine is brought down and then back up.

Update 01:03 AM PST: Spiderman was just brought down. It should be brought back up within the hour and another 30 minutes or so to get things configured on the new OS/install and hardware.

Update 02:28 AM PST: The final sync took a little longer than expected (by about 12-13 minutes). The machine is back up and running on the new hardware but still needs to be configured. Services should be restored within 30 minutes or so. Another update will be made when all services are restored.

Update 03:01 AM PST: Ok, all services should be restored on ‘spiderman’ if you are on spiderman and still having issues please contact support.


3:06 pm

MySQL Kernel Changes

Posted (July 7th, 2010 at 3:06 pm PST) by Leon

Over the next week or two, we will be changing the kernels on a few of our MySQL servers. The kernel change will require a reboot once the changes have been applied which will cause, roughly, a 10 minute downtime window for your MySQL services. These changes will begin tomorrow after 6:00 PM PDT and will be done in alphabetical order with the server names posted below.

bonito bubsy bullwinkle butterfish chapulin cod coelacanth dopey dumbo elmerfudd eltigre fatalbert felix flounder foghorn goldfish gourami grouper gumby guppy halibut herring inki jetsons koi leghorn mackerel magoo mahimahi minnow molly narwahl phillip popeye pumbaa salmon sardine seahorse shaggy slowpoke solomon spaceghost spongebob swordfish tarpon terrance tetra topcat towelie tuna wahoo yellowtail yogi zazu

If you have any questions, please contact our support staff!

Update 07/08/2010 8:21PM PDT: The following servers have now had their kernel upgraded and are running normally. We will continue with more upgrades tomorrow.

bonito bubsy bullwinkle butterfish chapulin

Update 07/09/2010 8:10PM PDT: The following servers are now upgraded and back up! More to come tomorrow…

cod coelacanth dopey dumbo elmerfudd

Update 07/14/2010 9:15PM PDT: We kinda fell behind! The following servers have had their kernel upgraded and are back up. More tomorrow!

eltigre fatalbert felix flounder foghorn goldfish gourami grouper gumby guppy

Update 07/16/2010 7:27PM PDT: The following servers now have their upgraded kernel and back up. Stay tuned for more!

halibut herring inki jetsons koi leghorn mackerel magoo mahimahi minnow

Update 07/28/2010 8:05PM PDT: Sorry for the delay but the following servers are now upgraded…

molly narwahl popeye pumbaa salmon sardine seahorse shaggy slowpoke

Update 08/03/2010 8:20PM PDT: The following servers are now upgraded…

solomon spaceghost spongebob swordfish tarpon tetra topcat tuna

Update 08/03/2010 9:20PM PDT: These servers are now upgraded…

wahoo yellowtail yogi zazu

Notification: The following servers that were part of the upgrade list were skipped due to issues and will be upgraded at a later date with another status post.

phillip terrance towelie


11:58 pm

Pinto in flames

Posted (July 3rd, 2010 at 11:58 pm PST) by Kyle

Pinto is having trouble recovering from a crash and we have been unable to get it back in online via remote hands.  An administrator has been dispatched to the datacenter to coax the machine back into business, we are sorry!

Pinto got a jump start and is wheeling it down the highway now. If you are still having issues with a site hosted on this machine please contact our support team from the web panel.


12:55 pm

Git upgrades

Posted (June 30th, 2010 at 12:55 pm PST) by AndrewF

We’ve just built packages for Git 1.7.1.1, which should be showing up on all Homie web servers within the next 24 – 48 hours. (You are probably in the Homie cluster if you signed up after June 2008, or if you’ve been moved to a new server within the past two years or so.)

If you are not using Git to manage your source code and/or other files, this upgrade will not have any effect on you whatsoever — Git is not a component of any packages we currently provide.

If you are using Git, have fun. :)


 
© 1996-2010, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
9:17 am

Kernel Upgrades

Posted (June 29th, 2010 at 9:17 am PST) by Kyle

This Saturday at midnight (PST) we will be upgrading the kernels on several of our shared hosting clusters: randy, postal, spacey, swarthy, looney, and blingy.  Each machine will be down for approximately 5 minutes while it reboots.  You can determine if your account is hosted on one of these machines by checking the Email Cluster on the Status tab of the Dreamhost Panel.

https://panel.dreamhost.com


8:39 am

Cassandra mysql hardware issues

Posted (June 29th, 2010 at 8:39 am PST) by Justin@DH

The cassandra mysql server started having some issues this morning, but came back up after a reboot. It seems to have gone down again now and is having trouble booting. An admin is already checking it out for hardware issues. We’ll try to get her back up and running as soon a possible. When we have more info we will post another update. Sorry for the downtime for customers on this mysql server.

— Update — The data (raid array) was transfered to another server and cassandra is back up. If you have any further issues related to this outage, please contact support.


12:48 am

Bandar Filesystem Check

Posted (June 29th, 2010 at 12:48 am PST) by JeffH

For everyone who bought stock in Filesystem Checking (NASDAQ sym. FSCK), today has been a real rally.  For those of you hosted on Bandar though, there’s going to be a bit of downtime while the linux filesystem checker works its magic.  We do apologize for the inconvenience, and will update this post when it finishes.

Update – 3:45pm PST: We’re actually in the process of putting up a new machine and copying backups over to this new hardware. With the issues this server has had over the past week or so, we feel it will be more stable on a new machine. Due to the amount of data housed on this server, both the FSCK and copying of backup data do take quite some time to complete. We are doing all that we can to resolve this issue ASAP, but unfortunately we’re at the mercy of how long these processes take. Again, we apologize for the inconvenience this has caused you.

UPDATE 6/30 5:30pm PDT The data copy is still going, and based on the transfer rate so far we estimate that it will take another six days to complete. During this time, the server is running from the last backup. You may keep uploading files without having to worry about getting them overwritten by the ones that are currently being copied over from the old machine. Sorry about the lengthiness of this process, and please let support know if there is anything else we can help you with in the meantime. We’ll update this post again when there is more progress.

UPDATE 7/4 2:30pm PDT Just a quick update to let you know that the data restore/copy over is at about 2/3rds done. We appreciate your patience during this process, and again, apologies for all the inconvenience this is causing.


8:24 pm

homie-vserver15 undergoing file system check

Posted (June 28th, 2010 at 8:24 pm PST) by Jason C

Customers on homie-vserver15 will notice that there are some issues reaching their sites right now. There was an unscheduled reboot performed by one of our techs that dovetailed into a file system check.  Since we don’t really advertise what host server a customer is on, here’s a list of affected Private Servers:

ps14114 ps14207 ps14219 ps14221 ps14307 ps14314 ps14419 ps14476 ps14782 ps14800 ps14982 ps14989 ps15056 ps15100 ps15101 ps16706 ps16707 ps17094 ps17271 ps20150 ps20433 ps21793 ps5155 ps5161 ps5162 ps5163 ps5165 ps5166 ps5173 ps5177 ps5189 ps5193 ps5196 ps5200 ps5530 ps5541 ps5578 ps5579 ps5583 ps5766 ps5771 ps6161 ps6307 ps6309 ps6310 ps6311 ps6370 ps7673 ps8129 ps8208

The “worst case scenario” estimate that our on duty admin has given for this file system check is 2 hours.  Hopefully it won’t take that long, but it’s always best to err on the side of Murphy’s Law.

We’ll have updates here as this issue progresses.

Filesystem check is finished, and machine is back up and running.


2:38 pm

Tehran server geting DDOS’d

Posted (June 28th, 2010 at 2:38 pm PST) by danwy

The server ‘tehran’ has been having some issues due to a DDoS attack that was targeting that server. We’ve already found the cause of the attack and have re-routed it. We had to give some services new IP addresses, so those changes will take a couple of hours to update everywhere. The cause of this issue has been resolved, but customers that have sites on the effected services may have to wait a little bit for the DNS change to propagate to them. We’re sorry for any inconvenience this causes you.


8:03 am

Email Server “smaug” issues with IMAP/POP3/SMTP/WebMail

Posted (June 28th, 2010 at 8:03 am PST) by patrickm

The email server “smaug” is currently having issues with IMAP/POP3/SMTP/WebMail.  Our Admin team is working with the problem now and hope to have it resolved quickly.

You can see which email server your account uses with the “Account Status” drop-down in your Web Panel.

We apologize for the inconvenience this causes.  Check back here for updates to this situation.  Thanks for your patience!

UPDATE 11:30am PDT: The mail outage earlier was due network connection issues to the file server holding the data. This has been resolved shortly after. However, one the servers handling authentication and sending got into a bit of an uncomfortable state, so we’re removing it from rotation, so you won’t get random errors until it’s fixed. We’re keeping an eye on the load, as now one less server is handling the flow temporarily. As we haven’t received any new reports for over 1.5 hrs, we consider this issue resolved. Sorry about the inconvenience this has caused.


5:16 pm

“Epiphone” being moved to new hardware.

Posted (June 25th, 2010 at 5:16 pm PST) by Leon

Unfortunately, it looks like the hardware that Epiphone is currently on has failed. We’re moving it’s data to new hardware and it should be back shortly.

Update 18:10: Machine’s coming up now, but it’s undergoing a file-system check. This can take a while to complete, so please be patient and we apologize for the inconvenience.

Update 18:58: Epiphone is back and ready to rock, even if only in a cover-band. If you’re still having problems, please contact your friendly DreamHost support roadie.


9:58 am

Moving server “gap” to new hardware

Posted (June 24th, 2010 at 9:58 am PST) by robert

The web server gap is currently experiencing issues with its hardware. We’re working on getting it moved onto new hardware.

Update: The server “gap” is on new hardware and all services have been restored. If you’re still experiencing issues, please contact support!


2:31 pm

swordfish getting reeled in

Posted (June 22nd, 2010 at 2:31 pm PST) by douglas

Well, swordfish mysql server has some interesting messages on it’s pretty, normally blue (now orange), LCD display that has us a little concerned. So we’re pulling the system out for a wee bit, do some poking around, swap out that RAM module it’s upset about, and then cast it back out into the great ocean of flashing lights with it’s other rack-buddies.

Update 2:54pm: Well gush durn it, we’d wun da lottera! The memory tested out just fine, and the ECC portion of the memory did it’s job, which the machine was happy enough to alert us that it had corrected a single bit error. Basically what can happen, and it’s very rare (but probable enough to warrant the extra expense of ECC ram in servers), is cosmic rays FROM SPACE!!! er.. ahem.. yeah well cosmic rays and other sources of potentially cancer causing ionizing radiation (my toy U-238 sphere I like to roll around, for instance, and even the carbon 14 and potassium in your body and bananas), can interact with silicon and electrons causing the charge to change at that junction, and thus cause a bit to flip– hence the single bit error (which ECC automatically detects, and corrects, so there is no data loss).

So yeah, very rare, but it can, and does, happen.

And so, I get to use one of the famous BOFH excuses. The cause was “Too much radiation coming from the soil.”


7:25 pm

Spacey file server issues causing slow sites & downtime

Posted (June 21st, 2010 at 7:25 pm PST) by Jason C

Users who are on machines that mount the file server “choker” may be seeing issues with their sites at the moment.  An admin is aware of the issue and is working at getting the misbehaving file server back up and running at optimal capacity.  Until then, if your sites are hosted on the servers listed below, there is a good chance that you will see some problems:

amp aquafina bambi cactuscooler cappuccino chai clamato coke culligan dasani downside eggnog endless evian fanta fetish1 fiji flipside fresca fuze geyser grapico hansens hiball ibc inko izze izzy koolaid lipton marmol milk minutemaid musclemilk nehi nesbitt nestea niagara oj pdxpeace pellegrino perrier planetapera plentymag ps4348 ps5130 qikserv001 redbull right riptide schweppes shasta silk slim sparkletts tab tampico tang tea tejava tizer tonic v8 voss welchs wishserver yoohoo zoot

Once we have an update from the admin working on this issue, I’ll make sure it is posted here.  Stay tuned!

Update: So it turns out that the problem was caused by a bad battery in the file server.  Fortunately, this has been resolved and the file server has finally finished rebooting.  If you’re still seeing problems with your sites, please contact support and we’ll do what we can to help you sort things out.


 
© 1996-2010, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
3:50 pm

Slow WordPress upgrades/installs

Posted (June 21st, 2010 at 3:50 pm PST) by danwy

With the new version of WordPress out, it looks like a lot of our customers are taking advantage of it and upgrading. The down side is that it has our one-click upgrade/install queue bursting at the seams. It is backed up at the moment and that is causing new WordPress one-click installs and upgrades to be delayed. It may take a few hours for your upgrade/install requests to be processed at this point. We’ll update this status post when the queue gets back down to normal, everybody will just need to be patient until it gets caught up.


8:41 am

SSH connection issue affecting servers

Posted (June 19th, 2010 at 8:41 am PST) by robert

Some of our servers are being hit by a SSH bruteforce attack. We’re actively working on a resolution.  Unfortunately this is still ongoing and I do not have a full list of servers affected.  If you are on one of the affected servers, you will experience issues with SSH connections.

UPDATE 2010-06-19 1:30 PM PST: Things have mostly calmed down. If you are still getting ssh errors such as “ssh_exchange_identification: Connection closed by remote host”, please contact support.


5:06 pm

“Bangkok” has some issues

Posted (June 18th, 2010 at 5:06 pm PST) by Leon

Looks like serverbangkok has lost connectivity. We have dispatched a tech to swap cables and get it back up. It should be back momentarily.

Edit: It’s back! Looks like a cable needed to be swapped. If you’re still having trouble, please vibe a good thought to your friendly DreamHost technical support guru.


4:53 am

jets shared server down

Posted (June 15th, 2010 at 4:53 am PST) by patrickm

We are also experiencing an issue with the ‘jets’ server, which is being addressed by our Admin team at this moment.  We expect to have this server back online shortly!

UPDATE 6:23am Pacific: This server is coming back up now, but requires a file-system check to verify the integrity of the data on the server.  Once this is complete, the server will be online again!

UPDATE 9:08am Pacific: The file system check had completed without error, but upon reboot the file system again showed errors requiring that the check be re-done. Our apologies for having to extend the downtime for this server again, we are working to get this back up as soon as possible but are limited to the disk check speed.

Update 5:10pm Pacific: Unfortunately, our attempts to correct the file-system errors did not go so well and the raid array is hosed. We are currently working on moving the machine and restoring from backups. We apologize for the inconvenience and hope to have the server up and running as soon as possible.


3:11 am

islamabad shared server down

Posted (June 15th, 2010 at 3:11 am PST) by patrickm

We apologize for the inconvenience this is causing.  We are working on getting this server back up now, but it is taking longer than expected.  We expect to have this issue resolved shortly!

UPDATE 2010-06-15 12:39 PM PST: The machine is back online. We had to replace its hardware. Unfortunately, there will be periods where the machine might have a lag. Since we will have to copy files from the backup machine to the local drive. The data restore may take a few days. FYI….your data is fully intact at the moment. Your home directory is mounted to a drive on the backup machine. Again, we sincerely apologize for any inconvenience this has caused and appreciate your continued patience. If you have any additional questions, please feel free to submit a support request.


12:54 pm

network issues — fixed

Posted (June 14th, 2010 at 12:54 pm PST) by douglas

So one of our cisco switches is having issues, and our network engineer is working on sorting it out.

Here is a list of affected servers:
buchalter lepke ironman castiglia karpis siegelprovenzano gambit escobar fay biggs cohen burke spilotro masseria moran greene hoffa dio storm kingpin greengoblin dillinger

Notice buchalter on that list? Yup we did too.

And affected PS:
ps11283 ps11550 ps11574 ps11944 ps11996 ps12637 ps12968 ps13010 ps13076 ps18573 ps19523 ps20112 ps21009 ps21503 ps22128 ps22766 ps4422 ps4431 ps4434 ps4446 ps4449 ps4451 ps4453 ps4455 ps4458 ps4459 ps4463 ps4674 ps4682 ps4812 ps4828 ps4856 ps4864 ps4915 ps4922 ps4933 ps5015 ps5047 ps5052 ps5065 ps5067 ps5069 ps5096 ps5101 ps5108 ps6784 ps8430 ps8443 ps14197 ps14204 ps14895 ps15197 ps15198 ps15199 ps15200 ps15203 ps15204 ps15809 ps26704 ps4743 ps4748 ps4752 ps4755 ps4762 ps4764 ps4770 ps4775 ps4778 ps4794 ps4803 ps4824 ps4833 ps4843 ps4851 ps4860 ps4874 ps5006 ps5012 ps5061 ps5079 ps5084 ps5087 ps5107 ps5110 ps6512 ps6808 ps6817 ps7727 ps7729 ps7869 ps8193 ps8231 ps8249 ps8429 ps8433 ps8442 ps10045 ps11179 ps11297 ps11589 ps11603 ps12197 ps12510 ps14218 ps14310 ps14311 ps21699 ps21771 ps22068 ps22083 ps7289 ps7304 ps7310 ps7311 ps7316 ps7317 ps7318 ps7324 ps7326 ps7332 ps7347 ps7412 ps7414 ps7505 ps7515 ps7520 ps7525 ps7528 ps7685 ps7689 ps7695 ps7696 ps7697 ps7724 ps7728 ps7740 ps7838 ps7839 ps7852 ps7854 ps8006 ps8014 ps8016 ps8051 ps8169 ps8238 ps8335 ps8340 ps8356 ps8434 ps9545 ps9547 ps9785 ps11125 ps11164 ps11318 ps11616 ps11717 ps12008 ps12011 ps12584 ps12920 ps12931 ps12966 ps13061 ps13062 ps13160 ps13994 ps13995 ps4477 ps4497 ps4505 ps4508 ps4517 ps4520 ps4524 ps4530 ps4532 ps4538 ps4540 ps4542 ps4545 ps4551 ps4696 ps4701 ps4783 ps4792 ps4815 ps4825 ps4831 ps4838 ps4861 ps4990 ps4996 ps5009 ps5077 ps5090 ps5744 ps5978 ps6159 ps8455 ps8716 ps8719 ps8744 ps9440 ps9762 ps10167 ps10940 ps11997 ps11998 ps12218 ps12316 ps13132 ps13163 ps13166 ps13934 ps14021 ps14022 ps14112 ps14387 ps14447 ps17178 ps17965 ps19067 ps22129 ps22147 ps22408 ps4481 ps4482 ps4494 ps4498 ps4501 ps4506 ps4512 ps4515 ps4521 ps4531 ps4534 ps4539 ps4541 ps4543 ps4700 ps4703 ps4737 ps4805 ps4826 ps4832 ps4841 ps4854 ps4994 ps8185 ps8187 ps8441 ps9498 ps9667 ps9878 ps11946 ps12015 ps12198 ps12512 ps12985 ps13158 ps13210 ps13233 ps13933 ps14070 ps14071 ps14385 ps14757 ps16765 ps17605 ps17970 ps18342 ps18354 ps18355 ps19682 ps21033 ps21234 ps21623 ps22752 ps4484 ps4492 ps4502 ps4507 ps4510 ps4513 ps4516 ps4529 ps4537 ps4544 ps4790 ps4807 ps4829 ps4836 ps4857 ps4927 ps4954 ps5020 ps5122 ps5630 ps5698 ps5723 ps6474 ps6527 ps6737 ps6754 ps7422 ps7475 ps7478 ps7479 ps7488 ps7498 ps7844 ps7947 ps8106 ps10034 ps10082 ps10087 ps10096 ps10100 ps10101 ps10104 ps10106 ps10109 ps10294 ps10296 ps10304 ps10308 ps10407 ps10410 ps10545 ps10768 ps10820 ps10998 ps11006 ps11255 ps13043 ps14148 ps14181 ps21412 ps9526 ps9549 ps9551 ps9561 ps9562 ps9563 ps9564 ps9565 ps9566 ps9567 ps9571 ps9578 ps9582 ps9587 ps9782 ps9818 ps9819 ps9825 ps9827 ps9839 ps9844 ps9944 ps9952 ps12608 ps12609 ps12962 ps13004 ps13050 ps13182 ps13221 ps14222 ps14408 ps14443 ps15844 ps18574 ps20114 ps20170 ps20315 ps21574 ps22079 ps22094 ps22116 ps23200 ps23201 ps23202 ps23653 ps23807 ps23983 ps24046 ps24105 ps24147 ps24584 ps25048 ps25694 ps25960 ps26314 ps26537 ps4746 ps4758 ps4761 ps4773 ps4806 ps4821 ps4849 ps4853 ps4882 ps5008 ps5014 ps5024 ps5070 ps5073 ps5076 ps5080 ps5082 ps5088 ps5092 ps5105 ps7686 ps8152 ps8154 ps8156 ps8158 ps8436 ps8742 ps8749 ps10364 ps15268 ps15269 ps19603 ps19605 ps19611 ps19637 ps19691 ps19754 ps20539 ps20604 ps21319 ps21382 ps21842 ps22241 ps22523 ps26554 ps7577 ps7614 ps7621 ps7629 ps7631 ps7645 ps7647 ps7653 ps7657 ps7669 ps7674 ps7694 ps7699 ps7711 ps7767 ps7931 ps7984 ps8115 ps8117 ps8165 ps8203 ps9700 ps9961 ps10639 ps10676 ps10746 ps10888 ps11350 ps11628 ps11968 ps13207 ps13216 ps14486 ps18603 ps18604 ps18605 ps20148 ps25660 ps25805 ps26053 ps26055 ps26059 ps26090 ps26294 ps26358 ps26555 ps4625 ps4627 ps4650 ps4652 ps4654 ps4664 ps4667 ps4673 ps4676 ps4814 ps4823 ps4830 ps4859 ps4888 ps4897 ps4903 ps4905 ps4993 ps5049 ps5054 ps5057 ps5059 ps5093 ps5222 ps6158 ps8271 ps10186 ps10189 ps10379 ps14131 ps14188 ps14354 ps14494 ps15127 ps15148 ps18484 ps18485 ps20426 ps20499 ps4633 ps4638 ps4651 ps4653 ps4666 ps4675 ps4810 ps4817 ps4842 ps4855 ps4868 ps4898 ps4904 ps4914 ps4918 ps4921 ps4931 ps5018 ps5036 ps5042 ps5043 ps5050 ps5053 ps5056 ps5063 ps5068 ps5071 ps5075 ps5089 ps5098 ps5102 ps5619 ps5623 ps5765 ps5770 ps5974 ps6144 ps6160 ps9841 ps11861 ps14014 ps14015 ps15147 ps17660 ps7582 ps7587 ps7594 ps7595 ps7597 ps7603 ps7605 ps7607 ps7610 ps7613 ps7626 ps7627 ps7649 ps7652 ps7656 ps7660 ps7661ps7718 ps7722 ps7935 ps7961 ps7971 ps7980 ps8054 ps8067 ps8181 ps8184 ps8188 ps8196 ps8308 ps8349 ps8435 ps10002 ps10215 ps10331 ps10361 ps10390 ps10391 ps10393 ps10659 ps10662 ps10670 ps10680 ps10694 ps10895 ps10896 ps10904 ps11148 ps11152 ps11262 ps11311 ps11347 ps11644 ps11651 ps11987 ps15191 ps15192 ps15193 ps15420 ps15484 ps15768 ps15769 ps18533 ps21861 ps22494 ps22722 ps9379 ps9381 ps9385 ps9386 ps9387 ps9388 ps9391 ps9392 ps9393 ps9395 ps9396 ps9397 ps9425 ps9435 ps9438 ps9442 ps9690 ps9696 ps9729 ps9731 ps9733 ps9735 ps9737 ps9755 ps9763 ps9953 ps9960 ps9988

Update: All fixed.. apparently the switch went bonkers and started sending out ARP requests on every interface… or a box stopped responding to ARP requests. Any case. it’s all better now.


12:00 pm

Buchalter is getting new hardware

Posted (June 14th, 2010 at 12:00 pm PST) by danwy

The Buchalter server has crashed due to some bad hardware and we are in the process of moving this server and its data over to a new machine. We are doing everything we can to make this a quick and seamless process. It will be down for a little while as this migration occurs. We will update this status post as soon as it’s back up and running on the new hardware.

Update: Our apologies for the delay. We originally thought the NIC had died on this machine, after further investigation on the switch we noticed a DDOS on the machine and we had our network engineer fix that up. The machine has not been moved to new hardware and is up and currently running.


11:00 pm

Network Maintenance 11 pm Sunday night, June 13 2010

Posted (June 13th, 2010 at 11:00 pm PST) by robertn

NOTE:  This will begin in 10 minutes

NOTE:  Maintenance is beginning with rows N1-4

NOTE:  completed all access switches, now reloading Distribution Switch — Almost done!

NOTE:  Done!


We will be continuing on the network upgrades this Sunday night with the switches in N row.  The servers that may experience a brief outage are:

|  algiers |  alouettes |  argonauts |  avengers |  baku |  bandar |  bayhawks |  bears |  beijing |  belmopan |  bengals |  bills |  blackhawks |  bluebombers |  bluejays |  bobcats |  bogota |  brasilia |  brewers |  bridgetown |  browns |  brumbies |  brussels |  buccaneers |  bucks |  buenosaires |  bujumbura |  bulls |  canberra |  cannons |  cardinals |  cavs |  chivas |  colts |  cowboys |  crew |  crusaders |  d-fenders |  dhaka |  dynamo |  eagles |  earthquakes |  eskimos |  expos |  falcons |  fever |  filesforever1 |  fire |  flames |  force |  gaborone |  grizzlies |  hawks |  heat |  herediano |  highlanders |  homie-vserver100 |  homie-vserver71 |  homie-vserver72 |  homie-vserver73 |  homie-vserver74 |  homie-vserver75 |  homie-vserver76 |  homie-vserver77 |  homie-vserver78 |  homie-vserver79 |  homie-vserver80 |  homie-vserver81 |  homie-vserver82 |  homie-vserver83 |  homie-vserver84 |  homie-vserver85 |  homie-vserver86 |  homie-vserver87 |  homie-vserver88 |  homie-vserver89 |  homie-vserver90 |  homie-vserver91 |  homie-vserver92 |  homie-vserver93 |  homie-vserver94 |  homie-vserver95 |  homie-vserver96 |  homie-vserver97 |  homie-vserver98 |  homie-vserver99 |  homie-vsql41 |  hurricanes |  hybridthree |  indians |  islanders |  jaguars |  jazz |  jets |  kabul |  lapaz |  liberty |  lightning |  lizards |  luanda |  manama |  marlins |  minsk |  monarchs |  nassau |  nationals |  ndjamena |  nets |  nuggets |  oilers |  olympiacos |  ottawa |  ouagadougou |  outlaws |  pacers |  packers |  panthers |  penguins |  phillies |  phnompenh |  pirates |  pistons |  porto-novo |  praia |  predators |  pride |  rangers |  rapids |  raptors |  ravens |  rays |  reds |  revolution |  rockets |  roughriders |  royals |  santiago |  saprissa |  sarajevo |  shock |  sofia |  sox |  spurs |  stampeders |  stjohns |  texans |  thimphu |  thrashers |  thunder |  tigercats |  tirane |  twins |  vienna |  vikings |  waratahs |  wizards |  yankees |  yaounde |  yerevan

Look for updates here.  As always, thank you for your patience as we strive to improve your Dreamhost Experience.


2:14 am

Cygnus raid trouble

Posted (June 12th, 2010 at 2:14 am PST) by Sandon

Unfortunately the machine ‘circinus’ is having some raid trouble. One of the disks was failing and it looks like the company that makes our servers messed up the ordering on the cables so the wrong drive was pulled putting it in a ‘failed’ state. The array should be recoverable but it can take some time to complete (30 minutes to an hour).

More updates to come as they are available.

Update: I apologize for the mistake the problem machine was cygnus not circinus.

Update 03:06 AM PST: The array has been recovered and the machine is back up and serving files.


10:37 pm

Hudson took a dump

Posted (June 10th, 2010 at 10:37 pm PST) by Sandon

The web-server ‘hudson’ was completely unresponsive and did not come back after a power-cycle. Investigation into the issue found that the machine will not power on and thus we will fail it over to new hardware. This machine uses NAS based storage so there should be no data loss moving to the new hardware and should be completed within one hour or less under normal circumstances.

Update 11:50 PM PST All services have been restored on hudson. If you are on ‘hudson’ and are still having problems with your sites please contact support.


 
© 1996-2010, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
11:00 pm

Network Maintenance 11 PM Sunday, June 6th – Monday, June 7th

Posted (June 6th, 2010 at 11:00 pm PST) by robertn

NOTE:  IOS WILL BEGIN IN 10 MINUTES  @11:30 PM

Note:  starting with rows m1-4 now.

Note:  All rows are done — now starting with distribution switch

Done!

This weekend we will be doing some traffic engineering and bgp work.  There will be no interruption of service while performing these tasks.

HOWEVER, we will be upgrading our IOS over  the next month.  This Sunday night around 11:30 PM  we will continue the process in our Downtown LA based data center.  This week M-row servers will experience a brief outage while those switches are reloaded.  We will post a warning here 15 minutes before we actually begin.  This will allow you to prepare for brief, intermittent outages over a 30 minute span.  Each outage should last only a few minutes (less than 5) and each server should not experience more than 2 brief outages as we reload the access and distribution switches.  The impacted servers will be:

homie-vserver22, homie-vserver23, penguin, magneto, venom, galactus, aquaman, hulk, homie-vserver26, homie-vserver27, homie-vserver28, riddler, mcgurn, homie-vserver01, castellano, homie-vserver19, homie-vserver20, homie-vserver21, homie-vserver37, homie-vserver139, joker, banshee, homie-vserver47, homie-vserver42, gambino, sugeknight, homie-vserver44, dhaccountbackup, iceman, nightwing, cyclops, homie-vserver45, spiderman, lansky, deadpool, homie-vserver48, pistone, brasco, homie-vserver31, gravano, genovese, homie-vserver32, homie-vserver15, catwoman, homie-vserver53, homie-vserver30, zwerbach, walsh, wolverine, thing, legs, terranova, floyd, bulger, fitzpatrick, coonan, homie-vserver18, homie-vserver17, homie-vserver16, homie-vserver29, homie-vserver56, mclean, spillane, robin, bugsy, nitti, homie-vserver25, homie-vserver24, colossus, cummiskey, kakuji, eastman, siegel, escobar, buchalter, homie-vserver14, lepke, ironman, homie-vserver10, homie-vserver09, gambit, homie-vserver08, karpis, fay, homie-vserver35, castiglia, homie-vserver55, homie-vserver07, provenzano, homie-vserver13, storm, dillinger, masseria, biggs, cohen, burke, spilotro, dio, hoffa, greengoblin, greene, homie-vserver34, homie-vserver33, moran, homie-vserver12, homie-vserver11, kingpin, homie-vserver46, trafficante, phantom, scalise, schultz, jubilee, giancana, mothman, capone, mangano, batman, cable, havok, hawkeye, dagger, cloak, gotti, rothstein, obelix, warmachine, sunspot, wolfsbane, rogue, supergirl, dalitz, bruno, bonanno, huntress, barker, marv, swampthing, cerebus, mekon, haddock, starfire, forge, thor, mahimahi, goldfish, snowbird, avalanche, sandman, bullseye, exodus, johnnyblaze, homie-vserver152, zatara, deathlok, spectre, quicksilver, professorx, apocalypse, gourami, molly, herring

Thank you for your patience.


11:12 pm

Singapore needs some lovin’

Posted (June 5th, 2010 at 11:12 pm PST) by brettg

Web server “Singapore” is being unresponsive to remote reboots. Our staff are currently on their way to move the machine to new hardware. If you have any questions please contact our support staff

Update 02:47AM: Sorry for the delay folks. Singapore has been up for awhile, but we were just verify all the services were completely operational before letting everyone know. If you have any issues please contact our support staff via the panel.


11:20 pm

Frisky cluster vserver hardware failure

Posted (June 4th, 2010 at 11:20 pm PST) by Sandon

One of our frisky cluster vserver (PS host hardware) servers has failed requiring us to fail it to new hardware. This process typically takes 1 hour to complete. This is effecting the following PSs:

ps6318 ps6541 ps7085 seawxhost

More updates to come.

Update 12:20 AM PST All service should be restored. Please contact support if you are still having problems and you are on one of the above PSs.


9:48 am

Debian Lenny on Homie Cluster

Posted (June 2nd, 2010 at 9:48 am PST) by Kyle

We are actively upgrading machines in our “homie” cluster from the Etch release of Debian Linux to Debian Lenny. This will be refreshing news for customers who have been anticipating the latest and greatest free software which Debian Lenny provides. Our upgrade process can be performed without service interruption except for the few minutes it takes to reboot the machine into the new release.

Things to keep in mind:

  • Customers using custom compiled software may need to rebuild it using the Lenny toolchain.

  • Support for PHP 4 has been dropped, as support for this version was terminated in 2007. Some applications and tools which used PHP 4 will be automatically switched to start using PHP 5, but others may need manual adjustments. Most recently released web applications should work correctly on PHP 5, though. For details, see the official end-of-life announcement and/or details on the changes in PHP5.


4:16 pm

Panel Downtime

Posted (June 1st, 2010 at 4:16 pm PST) by Leon

We’re currently experiencing issues with our web panel (panel.dreamhost.com) but are working on bringing it back up. Please check back for updates!

Update 4:50PM PDT: The panel is now back up and functioning normally. We apologize for any inconvenience this has caused you.


11:00 am

Frisky mail issues

Posted (May 31st, 2010 at 11:00 am PST) by andrea

We are experiencing some problems with our frisky mail cluster which resulted in unreliable mail access or sending or authentication both in webmail and non-web email clients. Our admins are working on this, and we’ll update this post as soon as we make progress.
If you’re not certain about which mail cluster you’re using, click on Account Status in the upper right corner of your panel, and look for your email server.

UPDATE 12:35pm PDT: The problem has been identified and being worked on. Sorry about the continued mail issues, we’ll post another update as soon as we can.

UPDATE 1:25pm PDT: We’re adding another machine into rotation to help the others handle the load while working on getting the problem server working again. We’ll update again soon.

UPDATE 5:10pm PDT: That new machine is added now, so this should help a ton. We’re leaving the status open on this to make sure things are working well! Will update when things are verified all clear!


7:46 am

Apus Server Downtime

Posted (May 31st, 2010 at 7:46 am PST) by Justin@DH

The apus webserver seems to have not been properly reporting that it was having a problem. We are looking into that issue, and an admin at our lax datacenter is looking at the server now. We are hoping it only needs a reboot but we will post further info as we get it. Our apologies for the longer downtime of this server.

— Update: The server was manually restarted from the admin at the datacenter and it is back up since a few hours. Sorry for the delayed update. We are keeping an eye on it for further issues.


11:00 pm

Network Maintenance Window: Sunday May 30th 11 pm until Monday May 31st 2 am

Posted (May 30th, 2010 at 11:00 pm PST) by robertn

Maintenance Completed.  Thank you for your patience.

Note:  All edge switches completed successfully.  Reloading Distribution switch

Note:  Starting edge switch reloads now.  Expect intermittent connectivity issues.

Note:  The IOS upgrade will be starting at 12:30 am.  This is your 15 minute warning!  (Well, actually 20 minute warning).

Note:  BGP work completed

This weekend we will be doing some traffic engineering and bgp work.  There will be no interruption of service while performing these tasks.

HOWEVER, we will be upgrading our IOS over  the next month.  This Memorial Day (approximately 12:30 am Monday May 31st)  we will begin the process in our Downtown LA based data center.  L-row servers will experience a brief outage while those switches are reloaded.  We will post a warning here 15 minutes before we actually begin.  This will allow you to prepare for a brief 10 minute outage.  The impacted servers will be:

homie-vsql29, tiny, garm, sprocket, muffy, barfolomew, sirius, hercules, ubu, courage, duchess, skip, yeller, kona, homie-vsql40, bishop, hawkman, homie-vserver40, sabretooth, mystique, hobgoblin, wonderwoman, preacher, homie-vserver54, homie-vserver51, homie-vserver39, homie-vserver36, homie-vserver52, homie-vserver49, homie-vserver50, homie-vserver41, homie-vserver38, homie-vsql28, homie-vsql26, homie-vsql27, joecool, hushpuppy, putney, argos, cerberus, krypto, clifford, naroga, duke, tramp, sui, underdog, fang, fluffy, petrelli, scarlet, greebo, menchit, sawdust, sebastian, spike, amber, tomoe, simba, odie, juno, lassie, mufasa, monty, scooby, snagglepuss, scratchy, midnight, demeter, neko, jarvis, heathcliff, growltiger, cinnamon, lionel, catbert, hushpad, leo, jones, snarf, jake, masevo, fwap, homie-vsql02, homie-vsql03, seabiscuit, grendel, lingling, barbaro, bennett, cheshire, buck, morris, puff, ein, cykotix, figaro, luna, garfield, guinness, dandilo, aslan, cassandra, mimsey, burbank, roary, orion, natty, sekhmet, cujo, bagheera, sifl, mau, bigglesworth, sparky, snowdrop, bill, milo, bingo, arlene, nala, homie-vsql07, homie-vsql04, homie-vsql06, bang, koi, homie-vsql05, homie-vsql08, homie-vsql09, sizzle, avarice, domination, watanabe, rhubarb, ren, constantine, dredd, angel, blade, shadow, humantorch, mrfantastic, ghostrider, luthor, wasp, cannonball, namor, silversurfer, spawn, greenlantern, tick, proty, twoface…

…which correspond to physical machines:

| mysql2453    | mysql1552  | mysql2062  | mysql2113  | mysql2276  | mysql2273  | mysql2112  | mysql1699  | mysql1551  | mysql1691  | mysql2060  | mysql1348  | mysql1545  | mysql2577  | mysql2309 | serf3072  | serf3071  | serf3134  | serf3070  | serf3069  | serf3068  | serf3067  | serf3105  | serf3117  | serf2596  | serf3100  | serf2520  | serf2597  | serf3098  | serf3099  | serf2519  | serf2911  | mysql2452 | mysql2449  | mysql2450  | mysql2275  | mysql2274  | mysql2451    | mysql1349  | mysql1452  | mysql1927  | mysql2858  | mysql2859 | mysql1347  | mysql1690 | mysql1698 | mysql1546  | mysql2061  | mysql1926  | mysql2860  | mysql0919  | mysql1227  | mysql1010  | mysql0966  | mysql1229  | mysql1062  | mysql1286
| mysql1155  | mysql1399  | mysql1061  | mysql1400  | mysql0809   | mysql1401  | mysql1063  | mysql0849  | mysql1402  | mysql1403  | mysql0920  | mysql1214   | mysql1086  | mysql1298  | mysql1087  | mysql0918   | mysql1220   | mysql1223   | mysql1216 | mysql1156 | mysql1231  | mysql1212  | mysql1221 | mysql1215  | mysql1159  | mysql2855   | mysql2856
| mysql2570  | mysql2571  | mysql2578 | mysql2918  | mysql2919  | mysql2572  | mysql2861  | mysql0640  | custom0612  | mysql0535  | mysql2920  | mysql2921  | mysql2854 | mysql1148   | mysql0773  | mysql0707    | mysql0805  | mysql0705   | mysql1305   | mysql0732  | mysql1084  | mysql1149     | mysql1162    | mysql1306    | mysql0922   | mysql1404    | mysql1233    | mysql1065   | mysql1217  | mysql1297   | mysql0847   | mysql1006   | mysql1477   | mysql1228    | mysql1157   | mysql1222    | mysql0808   | mysql1213   | mysql1224  | mysql1232  | mysql1003   | mysql1085    | mysql2915  | mysql2912  | mysql2916   | mysql2917   | mysql2383    | mysql2924    | mysql2926   | mysql2927  | mysql1230  | mysql2925   | mysql2923    | serf3007   | mysql1007    | mysql1158  | serf3103   | serf3102    | serf3096   | serf3082   | serf3164   | serf3083  | serf3089   | serf3084  | serf3101  | serf3088   | serf3085   | serf3087   | serf3086  | serf4571  | serf2784  | serf3107   | serf3160    | serf3162          |

Thank you for your patience while we improve your Dreamhost Experience.

If you experience an outage lasting longer than 10 minutes from the upcoming notification of this work commencing, please contact your Happy Dreamhost Support Representative!


12:45 pm

Thing Got Smashed

Posted (May 29th, 2010 at 12:45 pm PST) by Leon

The “thing” webserver is currently experiencing issues with its hardware and we’re working on bringing it back up. We currently have no ETA but are diligently working to bring it back!

Update 2:13PM PDT: Thing is now back up on new hardware! All services on the machine are restored without any data loss! If you’re on “thing” and still experiencing issues, please contact support.


12:35 am

Pyongyang went bellyup

Posted (May 29th, 2010 at 12:35 am PST) by Sandon

The web-server pyongyang has had a hardware failure where it no longer POSTs. We are failing it over to a new set of hardware (moving the disks from the old machine). There should be no data loss during this process and it typically takes approximately one hour.

Update 01:23 AM PST: The machine has been moved to new hardware and is back up and running with all services restored. If you are still having issues with your sites and you are on pyongyang please contact support.


 
© 1996-2010, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
12:02 am

Robin laid an egg

Posted (May 29th, 2010 at 12:02 am PST) by Sandon

The web-server ‘Robin’ is suffering from some 3ware sickness where it refuses to rebuild its array after a disk has failed but is otherwise performing normally without problems when it is left degraded. Because we can’t leave the machine in a degraded state we have decided to give it a dose of Areca and migrate it to an entire new set of hardware.

In the next hour the machine will be going down for what could be up to an hour while the final sync of all user data is done to insure all data has been moved to the new server and services get restored on the new hardware. There should be no files or data lost during this transition but the server will be inaccessible during the final sync to insure no files are created while the sync is being ran and everything is copied correctly. An update to this post will be made when the transition starts and finishes.

Update 01:33 AM PST This got a bit delayed due to trouble with pyongyang but I took the machine down just now to start the migration process.

Update 03:45 AM PST: The final sync took a bit longer than expected but otherwise there haven’t been any problems. All data has been synced and now the services are being configured on the new hardware. ETA is about 15 minutes until services are restored.

Update 04:35 AM PST: Unfortunately there were a couple setbacks during the final setup of the server which caused some delays in getting it back up and running but the server is now back up and all services are restored. Please contact support if you are still having any problems and are hosted on ‘robin.’


4:45 am

nuggets drive check in progress

Posted (May 28th, 2010 at 4:45 am PST) by Ken

Due to filesystem errors causing the drive to go into readonly mode, we are currently running an fsck on the main drive. Thank you for your patience while we work on this issue.

Update 12:02pm:  Filesystem check complete, drives remounted, and everything should be good now.  Thanks for your patience, and if you have any further issues, please contact support!


5:21 am

Belair lost it’s oomph – RESOLVED

Posted (May 27th, 2010 at 5:21 am PST) by Ken

Server belair lost it’s power supply. We are in the process of moving all services to new hardware and appreciate your patience while we resolve this issue.

UPDATE 06:21 PST: Belair has been moved to new hardware. If you continue to experience issues, please contact technical support through normal channels.


10:10 am

Looney filer problems

Posted (May 26th, 2010 at 10:10 am PST) by danwy

We are currently experiencing some issues with our looney server cluster. One of the file servers that services this cluster has become unresponsive and is causing issues on all the servers in the cluster that access it. Here is list of web servers that are affected by this:

apok axl blanka brazil charm curly dhalsim egon guile limbo-looney moe slappy soy tank tofu vega winston

It will also cause problems with mail service on this cluster as well. To see if you’re on the looney mail cluster, go to the ‘Custom MX’ section of your web panel, select a domain and scroll to the bottom. Our admins are working on resolving this and should have it back to normal shortly. We will update this post when the issue is fixed.

Update – 11:00am (PST): This issue should be resolved now. The fileserver recovered from the problems it was having earlier and we’ve rebooted all the servers in the cluster that use it. Mail and web services should be restored to all affected servers at this point. If you’re still having issues, please contact support and they’ll be happy to assist you further.


4:07 pm

Roundcube upgrade tonight at 9PM PST

Posted (May 25th, 2010 at 4:07 pm PST) by jordan

Tonight we will be rolling out the latest stable release of roundcube 0.3.1. This upgrade should not cause any downtime but some customers may be logged out and have to log back in. The purpose of this upgrade is to patch security holes found in the old version and all of the new features roundcube 0.3.1 provides which can be found here: http://trac.roundcube.net/wiki/Changelog . After the upgrade is complete we will update this post with more details.

Update 09:32 PM PST – The upgrade is complete. You may notice that if you have a custom login image set here https://panel.dreamhost.com/index.cgi?tree=mail.webmail& , it now works with roundcube! Please contact technical support with any issues you are having with this new version of roundcube.

Update 12:36 AM PST – After the upgrade it appears some users were having problems seeing their address books / sending email when logging in with only the first part of their email address (eg. jordan instead of jordan@domain.com) . This was caused by a configuration error and has been resolved. We apologize for the trouble and if you are still experiencing problems viewing your address book or sending email please contact support an we will be happy to assist.


1:13 pm

Webmail problems

Posted (May 25th, 2010 at 1:13 pm PST) by danwy

Several of our webmail machines currently have a high load and is causing webmail to fail when loading. We’re aware of the situation and are working on resolving it. We’ll update this post when we have more info.

Update – 1:45pm PST: This issue should now be resolved. Unfortunately, we were not able to find what caused the load spike, but things are much better now. We’ll continue to monitor the situation. If you’re still having problems with webmail, please contact support.


4:28 pm

Pogo Moving to New Hardware

Posted (May 22nd, 2010 at 4:28 pm PST) by Leon

The pogo web server is currently experiencing issues with its hardware and we’re working on moving it. We expect it to be back up within the next hour but please check back for updates.

Update 05:30 PM PDT: Pogo is now back on new hardware and serving your websites again! If you’re on pogo and still experiencing issues, please contact support.


10:35 pm

Maseru Hardware Failure

Posted (May 21st, 2010 at 10:35 pm PST) by Sandon

The web-server ‘Maseru’ has been crashing and randomly rebooting itself suggesting a hardware issue. The drives and raid controller seem fine so we will be moving the disks over to a new set of hardware to fix up the problem. ETA is 30 minutes to 1 hour.

Update 11:23 PM PST: Maseru is back up and running on brand spanking new hardware (minus the raid controller and drives). No cutsomer data should have been lost during this process and all services are back to normal. If you are still having problems with your sites and you are on ‘maseru’ please contact support.


7:31 am

Web Panel Downtime — DreamHost.com DDOS

Posted (May 20th, 2010 at 7:31 am PST) by robert

We are currently experiencing an issue with the customer Web Panel which is causing extreme slowness and downtime.  We are working to resolve this issue now and will update this post as soon as we have more information.

Update 08:00 AM PST: Everything except dreamhost.com should be back up. There was a DDoS attack to dreamhost.com which caused our load balancer to get a bit overwhelmed which also took down some other services. We are working on restoring access to dreamhost.com right now and more updates to come.

Update 12:05 PM PST: We’re in the process of separating the handling of various services run off of the DreamHost.com domain to allow us more control and stability (this will help us minimize the fallout from any future attacks and with recovering from this one).

Update 15:20 PM PST: The attack has dropped down to manageable levels, and our web site should be visible to most users now.

Update 13:32 PM PST (Friday): A post-mortem analysis of the situation is now available on our main weblog: http://blog.dreamhost.com/2010/05/21/yeah-about-that-downtime/


12:55 pm

homie-vserver71 RAID failed (psych!)

Posted (May 19th, 2010 at 12:55 pm PST) by douglas

Well, something very unusual happened to homie-vserver71, two drives in the same disk group failed, which means the RAID array went with it.

We are working to restore your machine from backups.

Affected guests:

ps12644 ps12652 ps12657 ps12663 ps12672 ps12693 ps12697 ps12706 ps12709
ps12712 ps12715 ps12718 ps12720 ps12723 ps12730 ps12733 ps12737 ps12741
ps12745 ps12750 ps12759 ps12767 ps12772 ps12774 ps12779 ps12799 ps12803
ps12807 ps12821 ps12827 ps12839 ps12847 ps12856 ps12882 ps12894 ps14196
ps20330 ps20347 ps20441 ps20447 ps20455 ps20510 ps20520 ps20620 ps20801 ps20881
ps21071 ps21250 ps21383 ps21418 ps21466 ps21847 ps21942 ps21991 ps22016 ps22099
ps22354 ps23029 ps23030 ps23031 ps23032 ps23162 ps23249 ps24083 ps24137 ps24221
ps24264 ps24343 ps24384 ps24502 ps24747 ps24756 ps24817 ps25086 ps25089 ps25106

And no others ;)

Update: Well it looks like the LSI card just went stupid and ejected both drives from the array. We re-added them, and so far everything looks fine and dandy. Whew! We’ll be monitoring the machine closely however.


 
© 1996-2010, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
8:00 pm

Additional Panel Database Maintenance, Tuesday, May 18 8PM PDT

Posted (May 18th, 2010 at 8:00 pm PST) by DreamHost

Just like Monday, we’ll be performing maintenance on our central database a second time next week, beginning Tuesday, May 18 8:00PM PDT, ending 11:59PM PDT. This will affect the following services:

DreamHost Webpanel
DreamHost-provided Webmail
Signups
Domain Registrations
DreamHost Wiki

Update, Tue May 18 23:48:10 PDT 2010: Delays in the maintenance process have pushed the finish time back a couple hours. Current estimate is 3:00AM PDT.

Update, Wed May 19 04:04:00 PDT 2010: This process has now been completed and the Panel is up and running again!  If you’re having any issues, please contact support!


8:00 pm

Panel Database Maintenance, Monday, May 17 8PM PDT

Posted (May 17th, 2010 at 8:00 pm PST) by DreamHost

We will be performing maintenance on our central database next week, beginning Monday, May 17 8:00PM PDT, ending 11:59PM PDT. This will affect the following services:

DreamHost Webpanel
DreamHost-provided Webmail
Signups
Domain Registrations
DreamHost Wiki


3:34 pm

Django 1.2 arriving soon

Posted (May 17th, 2010 at 3:34 pm PST) by AndrewF

Those of you who are using Django may be interested to hear that version 1.2 has been released, and will be deployed on the majority of DreamHost’s servers in due time (probably in two to three weeks — I’ll make an additional announcement when that time comes). Information on changes in Django 1.2 is available at:

http://docs.djangoproject.com/en/dev/releases/1.2/

If you run a Django application on DreamHost’s servers, please read and acquaint yourself with these changes, as Django 1.2 introduces some backwards-incompatible changes. If you do not run any Django applications, this update will not affect you in any way whatsoever.

Update: This will probably be Django 1.2.1, owing to some issues with 1.2.0 which were discovered at the last minute.


7:14 am

Customer Web Panel Downtime – UPDATED

Posted (May 17th, 2010 at 7:14 am PST) by patrickm

We’re very sorry to report that our Customer Web Panel is down at this time.  We are working to restore service at this time and will update this space when the issue has been resolved!  Apologies for the inconvenience!

Quick update: We did *not* start the maintenance earlier than scheduled and advertised in the above posts. We were having technical difficulties. The panel is now operational, and we’re looking into what caused the earlier downtime. Sorry again about the inconvenience.

Last update: The cause of the downtime was overloaded web services in the panel. We’re looking into a permanent solution, including the scheduled maintenance later today and tomorrow.


3:36 am

shared server chisinau down for maintenance

Posted (May 17th, 2010 at 3:36 am PST) by patrickm

We sincerely apologize for the downtime, as the chisinau server had to be taken down for emergency maintenance.  A simple reboot of the server was not working, so we dispatched an engineer to the data center.  The server is being particularly difficult in coming back up, so it may be a little while longer before service is restored.  We will update this space when more info is available!


11:00 pm

Network Maintenance Window — Sunday May 16th, 23:00

Posted (May 16th, 2010 at 11:00 pm PST) by robertn

Monday, May 17th 2010 01:39 — this maintenance window is now closed.  All tasks attempted were completed successfully.  Look forward to more improvements next Sunday!

There will be a network maintenance window this Sunday night/Monday morning from 23:00 – 02:00.  We will be cleaning up cabling, standardizing configs, and doing some miscellaneous spring cleaning.

This work should have no impact on customer connectivity.

That being said, the caveat of “there is always a chance…” is in force.  If you DO experience a problem during this window, please call your Happy Dreamhost Customer Support Representative immediately.

As always, thank you for your patience and your patronage.


2:17 pm

homie-vserver159 & 162 currently having problems.

Posted (May 14th, 2010 at 2:17 pm PST) by Ralph

We are currently experiencing problems with one of our vserver hosts that are administration team is actively working on. PS guests affected by this are:

ps23345 ps23364 ps23388 ps23431 ps23461 ps23487 ps23517 ps23553 ps23634 ps23668 ps23705 ps23738 ps23777 ps23827 ps23863 ps23908 ps24065 ps24120 ps24162 ps24209 ps24252 ps24294 ps24331 ps24370 ps24679 ps24812 ps24887 ps24890 ps24896 ps24936 ps24940 ps24995 ps25240 ps25289 ps25326 ps25368 ps25382 ps25411 ps25498 ps25501 ps25511 ps25530 ps25576 ps25597 ps25599

We are having issues with the kernel and the network on the machine. As soon as we have further information we will post an update here. Sorry for the problems.

Update 15:28: Additionally, homie-vserver162 is seeing the same issue. This affects the following PSes:

ps23349 ps23391 ps23434 ps23464 ps23490 ps23520 ps23556 ps23592 ps23638 ps23670 ps23709 ps23741 ps23780 ps23830 ps23866 ps23911 ps23954 ps24001 ps24068 ps24123 ps24165 ps24212 ps24255 ps24297 ps24334 ps24376 ps24650 ps24665 ps24676 ps24685 ps24777 ps24780 ps24809 ps24814 ps24892 ps24916 ps25004 ps25039 ps25041 ps25066 ps25068 ps25087 ps25130 ps25142 ps25149 ps25228 ps25274 ps25279 ps25296 ps25412 ps25500 ps25533 ps25537 ps25548 ps25589 ps25636 ps25665

We are continuing to look into the issue.

Update 17:25: This has been resolved for these two machines for a while now, but there was a larger issue that needed resolving as well. Both are done now. Please write to your happy DreamHost support fellow/lady and they’ll be happy to help you further.


2:55 am

Scalise downtime

Posted (May 14th, 2010 at 2:55 am PST) by Sandon

The web-server ‘scalise’ waas getting some file-system errros so it had to be brought down to run a fsck on it. This process takes anywhere from 2-5 hours depending on the amount of files on the machine. More updates to come.

Update 06:42 AM PST: From the pass its on it typically means the fsck is about 70% done at this point. More updates to come.

Update 010:00 AM PST: Done and happy again.  Thanks for your patience.  Let us know if you’re still having any problems.


5:48 am

bishop is taking a sabatical

Posted (May 13th, 2010 at 5:48 am PST) by Ken

bishop is going to be down for a while to fsck the /home directory. This may take up to several hours to complete.

We thank you for your patience while we work on this issue and will update this post as soon as we find out more.

Update: Bishop’s fsck of it’s /home directory has completed and all services are back up. Please contact our support staff if you’re noticing any further issues.


5:29 am

SMTP issues (RESOLVED)

Posted (May 12th, 2010 at 5:29 am PST) by patrickm

We apologize for the current issues with outgoing mail.  We are working with our Admin team to resolve this issue and hope to have service restored quickly.  Please check this space for updates.

UPDATE: This issue has now been resolved!  We found a problem with our internal scripts which was preventing us from pushing changes to the mail servers.  Now that we have identified and fixed this issue, we can prevent it from happening again.  Outgoing mail service has been restored, but please contact support if you still have any issues!

UPDATE: We’re still working on this issue. It seems that the fix we pushed out before didn’t hold up. We’ll update this post when we have more info.

UPDATE – 11:30pm PST: We believe that the issues on our end causing mail problems are resolved now. There were actually a couple of different issues, one of which we don’t have much control over.

The first issue was due to one of our mail db’s on the backend. This was causing mail connections to our outgoing mail servers to hang and would result in a 450/451 configuration error when trying to send mail. We have put a temporary fix in place while we resolve the root of this problem, so this error shouldn’t be popping up any more.

The other issue is mail timing out when trying to send. We believe that this is due to several ISP’s starting to block port 25 at some point over the last 24 hours. ISP’s frequently block this port to help prevent spam from being sent through their system. We tested several customers email addresses via multiple ISP’s and found that changing the outgoing mail port from 25 to 587 in the email client fixed the issue in every case. We can’t speculate as to why this all of a sudden started today, but confirmed on dozens of tests that we ran that the above changed resolved the issue. We’ve done extensive testing on our end to insure that this wasn’t caused by a problem on our end. If you’re still having problems after making this change, or don’t know how to change this setting, please contact support so that they can assist you further.


 
© 1996-2010, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
4:00 pm

Galaga outta guys

Posted (May 10th, 2010 at 4:00 pm PST) by Leon

The power supply for this machine died and is being replaced. It should be back shortly.

Update 4:14: Galaga is back up now. Should you find yourself stuck in a tractor beam, please contact one of our friendly DreamHost techbots for assistance.


8:34 am

Network issues with 2 racks in LAX datacenter

Posted (May 9th, 2010 at 8:34 am PST) by Justin@DH

Our network admin is already on the job and checking out why there is some networking issues to the following machines. We will do everything we can to get this issue resolved as soon as possible. When we have further info we will update this post.

abuja amsterdam apia asuncion belgrade bucharest castries dakar doha homie-vserver153 homie-vserver154 homie-vserver155 homie-vserver156 homie-vserver157 islamabad kathmandu kigali kingstown lima lisbon managua manila melekeok moscow muscat niamey nouakchott oslo rangoon riyadh victoria warsaw wellington windhoek yangon

UPDATE: The cause of the problem was found, a massive attack to a domain on one of the above servers, which caused 4x the traffic which saturated the uplinks from those racks. The domain has been dealt with, and the ip being attacked blocked. We are continuing to monitor the situation to make sure there were no other causes.

Sorry again for the downtime and problems caused by this issue. If you still have issues and you are on the above servers, please contact support and we would be happy to help.


10:32 am

Webmail currently down

Posted (May 7th, 2010 at 10:32 am PST) by Ralph

Webmail service is currently down. We are working on the issue and will have it resolved as soon as possible. Once it is back we will update this post with further information.

Update: 10:49AM Pacific: Webmail service is restored. We are still watching the situation and determining the cause. If you do notice problems with webmail still please contact support. We are currently testing webmail on current open support cases. So far so good!


2:35 am

outlaws had a fight with the inlaws

Posted (May 6th, 2010 at 2:35 am PST) by Ken

outlaws is currently experiencing an FSCK disk check. This can take up to several hours to complete.

We thank you for your patience while we fix this issue and will update this post as soon as we have more information.

Update 07:24 AM PST The FSCK has finished and all services have been restored


9:59 am

Issues with email access

Posted (May 5th, 2010 at 9:59 am PST) by danwy

We’re currently experiencing some high loads on some of our email machines. This is causing login errors and slow access to email (including webmail). We don’t have an ETA yet on when this will be resolved, but our admins are aware of the situation and are working on reducing the loads on these servers. We will update this post as soon as we have any further information.

Update – 11:50am: We found the cause of these issues and are in the process of fixing it now. Our mail load balancer was piling up all the incoming connections on to only a couple of servers. As a result, those servers became overloaded and were taking a long time to process requests. In some cases, timing out completely. We’ve got things under control again and will be making changes on our backend to prevent this from happening again. If you continue to have issues with your mail, please contact support for further assistance.


2:59 am

Eskimos has frozen

Posted (May 5th, 2010 at 2:59 am PST) by Sandon

The machine crashed after the raid array became degraded and upon rebooting the file-system went into read-only mode so we had to take the machine down into single user mode to run an fsck (file-system check).

This process has been known to take anywhere from 2-5 hours depending on the amount of data on the server. More info to come as it is available.

Update 06:11 AM PST: The fsck has finished the first pass which usually means it is atleast 50% completed.

Update 07:00 AM PST: Unfortunately the raid controller took a dump while the FSCK was running. We have replaced the controller with a new one but the fsck must now be restarted.

Update 2:14 PM PST: fsck has finally finished, and the machine rebooted.

Unfortunately looks like there’s another hardware issue. We’re looking into it now.

Update 4:15 PM PST: It appears the disk array it’s self was corrupted by the bad RAID card, we are now preparing to restore your data from backups.

Update 6:15 PM PST: We are currently preparing new hardware for the machine to be moved to. While data is being copied over, sites will run from backup. You may update your data as often as you wish, as the syncing process won’t override new information. We don’t have an estimate on how long the data move will take, but please check back here for updates. We apologize for any inconvenience this is causing.

Update 9:10PM PST: The hardware has been in for awhile now, and services are coming up as part of the fail-over process, but configs are still being run.

From the comments, some people are concerned about how long it will take to restore the data, and about data loss. Basically anything on you mysql server is fine, any new data uploaded to your server after nightly backups were run would not be available, nor would any directory that matched the backups exclusion pattern (“backup” and various iterations of that– such backups aren’t allowed under unlimited plan anyway). The time to move all the data back into local storage will take some time (probably a week), but, in the meantime, you sites will run fine from the remote storage; however, you may experience a reduction in site performance during that time.

Update 12:20PM PST: New hardware still up and running, but the backup machine had a degraded array (affects performance, but not your data’s integrity) and it’s just a case of  bad timing, unfortunately.  This array is rebuilding, but in the mean time, availability of the server may vary.  We apologize for the downtime, and have discussed ways to improve this process in the future.  If you have any issues not related to Eskimos, please contact support.

Update May 6 3:16 PST High load on the server forced us to restart. Please be kind to the server.

Update May 6 4:30 PM PST The server is stable now with load staying around 10, with occasional spikes as high as 28. We’ll be good here on out, so long as there isn’t another mad rush on the server.

Update May 11 We’ve reached the 75% point on moving data from remote storage to local storage. All data will be back onto local storage in 2 days at the current rate.


2:09 am

dunker has been dunked

Posted (May 5th, 2010 at 2:09 am PST) by Sandon

The file-server named ‘dunker’ is having a problem coming back up after a maintenance reboot. This is causing high loads on some of our ‘frisky’ cluster servers and some customers sites to not be accessible. While not effecting all iit is effecting some sites on the following servers:

allbusiness atgaum athena barracuda beaker beast belair bilbo bloghogwarts boxster brompton brweber2 bsmovies bugatti bumper cadillac camaro carlititos carrera cayman chevelle continental corvette cutlass dart davidmihm delorean downtime ecsos edsel elcamino eldorado elise enzo erinyes esprit ewebice fairlane favre ferrari fiat fiero firebird galaxie geekoracle geep george gibs gigantiq gmais gremlin highground hornet hotweb hudson hummer hupmobile ikero impala jaguar jetta journal kbooradio kemono lancer laragmag learnwake lesabre lifehappy llhost lobster longhollow madromsrevenge malibu manmeat markets mastertech mclaren merkur metaweb miata model-t momentor montecarlo murcielago netthru ninjoomlahost noddydog norice nova odin ohnonotagain olds pacer packard passat phaeton pierce pinto pione probe prodlinux0 prowler ps13568 ps21601 ps3778 ps4203 ps4227 ps4234 ps4327 ps4573 ps4680 ps4735 ps5160 ps5213 ps5516 ps5521 ps5609 ps5917 ps6123 ps6303 ps6318 ps6529 ps6541 ps6966 ps7085 ps7148 ps7684 ps7883 pulp pulsar quioxte radix rambler rawstock renault riadev riviera rolls scion seawxhost senumero sexylinker shatil shelby show-links sindhu skoda skylark slithercunt softjei spiff spsm1 starrynight stingray stratus studebaker submerge swervin swifty t3ti tatummarketing tempest tesla theimagedepot thog thunderbird thunderfury tiltneck tinydancer titan todocorazon touareg towncar trans-am travelinsync unisoft up2008 valhalla veyron viewgenie vs-rachelleb wiiplayer wrangler

More updates to come as they are available.

UPDATE 04:18 May 05 PDT: dunker has come back up for air. If you continue to have problems with one of the affected servers, please contact Technical Support via normal channels.

We thank you for your patience while we resolve these issues.


9:07 am

Domains using PHP4 to be upgraded to PHP5 in two weeks

Posted (May 4th, 2010 at 9:07 am PST) by Justin@DH

As previously posted, PHP 4 is no longer supported and is lately becoming incompatible with our other offerings, such as mysql server changes to support PHP 5.3. The mysql upgrades are around 25% complete, and most downtime we have seen in support has been from customer using modern software that supports PHP 5, but inadvertently using PHP 4 on the domain from years ago when they first set it up. In two weeks, our developers will upgrade domains setup to use PHP 4 to instead use PHP 5, to avoid possible downtime caused by this issue.

What does this mean for you?
– Customers that signed up in the last few years are already using PHP 5 and will see no changes or downtime.
– Customers who have some domains setup with PHP 4 and are running modern software (anything a few years and newer should be just fine) will not notice that anything was changed, and in fact this change will prevent them having downtime in the future when mysql upgrades are complete.
– Customers who are using very old software that requires PHP 4 or software that needs register_globals* on, will need to upgrade to more modern software. Whether it is the mysql upgrades or the PHP version change, this older software will eventually no longer be compatible with our hosting.

This change is being made for shared hosting customers only. Dedicated server customers are not being upgraded at this time and will be contacted directly about upgrades in the future when it is their turn.

* register_globals – PHP Manual: “This feature has been DEPRECATED as of PHP 5.3.0. Relying on this feature is highly discouraged.”


7:57 pm

Kuwaitcity Hardware Issues

Posted (May 2nd, 2010 at 7:57 pm PST) by Leon

The kuwaitcity web server is currently experiencing issues and we’re working on moving it to new hardware. We expect to have it back up within an hour but please continue checking for updates.

Update 10:32PM PDT: We apologize that the swap took longer than expected but we ran to a bit of issues with the new hardware, so a bit more “swapping” was needed. In any case, kuwaitcity should now be back up and running! If you’re still experiencing issues and are on kuwaitcity, please contact our support team!


8:31 am

Cadillac hardware issues

Posted (May 1st, 2010 at 8:31 am PST) by robert

The web server Cadillac is currently experiencing issues with its hardware upon reboot. We’re working on getting it moved onto new hardware.

Update 9:30AM PDT: Cadillac is on new hardware and all services have been restored. If you’re still experiencing issues, please contact support!


 
© 1996-2010, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
4:19 am

charvel forgot where it keeps things

Posted (April 30th, 2010 at 4:19 am PST) by Ken

charvel is have a raid issue. We are currently looking into it and will post more information as we find out.

Thank you for your patience while we look into this problem.

UPDATE 30 April 2010 0521PST: charvel is now up. Thank you for your patience while we strive to improve your Dreamhost experience.


11:29 am

HVAC issues at LAX datacenter

Posted (April 29th, 2010 at 11:29 am PST) by kitchen

Our LAX datacenter is currently experiencing some HVAC issues. The datacenter operators are working on resolving the issue currently, and they expect to have things back to normal in the next 30 minutes. At this time we are not experiencing any issues other than a number of temperature warnings, but we thought we should keep everyone abreast of the situation.

We’ll post back here with an update once things are back to normal!

Update 11:45am pacific:

We just got notice from our datacenter operator that they’ve resolved the HVAC issues and temperatures should be coming back down shortly. We’ll continue to keep an eye on the situation and make sure everything is running smoothly.

Update 1:48pm pacific:

Everything is not running smoothly.  Lisbon, Managua, Suba, Wellington, and couple of dedicated servers are currently being worked on to bring them back up after this morning’s issue.  Will update with details as they become available.  Thanks for your patience.    Please contact support if you are having issues and are not on one of the above servers.

Update 4:28 PM PDT: We apologize for the length of time it took to resolve; however, Lisbon, Managua, and Wellington are now back up and running. If you’re still experiencing any issues, please contact our support team.


3:45 am

Homie cluster email issues – RESOLVED

Posted (April 28th, 2010 at 3:45 am PST) by patrickm

We’re very sorry to report that we are experiencing an outage with email on our Homie cluster.  To see which cluster your account is under, please check the “Account Status” drop-down in your Web Panel and check under Your Email Server and if it starts with “homiemail” you are in the affected cluster.

We hope to have this issue resolved as quickly as possible!  We will update this space once we have more information or a resolution!

Update 4:14 am – This issue is now resolved.  We have collected a lot of information about this problem and we’re fairly certain we have the cause locked down.  This will allow us to make changes that should prevent this from happening again.  In a nutshell, database updates that happen around that time are causing problems for the email servers.  Please contact support if you’re still experiencing any issues with your email.


10:29 am

MySQL Server Software Upgrades

Posted (April 27th, 2010 at 10:29 am PST) by Justin@DH

This is a general notice of changes happening on our mysql servers. All servers have already had their operating system upgraded, in preparation for two big changes:
* The upgrade from MySQL 5.0 to 5.1.
* The change from “old password format” (pre-4.1) to the new standard password format.

The first change will allow us to offer trigger support in the future, because a new privilege specifically for triggers was added in 5.1. The second change will allow us to support and offer PHP 5.3 in the future, because the mysqlnd library in PHP 5.3 does not support old password format. When these changes are implemented over the next month or two, we can start to work on those two commonly requested features/changes.

For the vast majority of customers, these changes are transparent and you will see no change in how your site functions. There is one case that can come up if using PHP 4. We haven’t offered PHP 4 for a long time , which is good because by default it does not support the new password format. For those with very old software that still requires this very old unsupported version of PHP, your time to upgrade to more modern software is getting close. If you have been a customer with us for more than a few years, and are concerned any of your sites are using PHP 4, you can check it yourself or contact support with a list of the domains. Also, if you believe you have any database problems related to an upgrade, please contact support.


7:47 am

pyongyang downtime

Posted (April 27th, 2010 at 7:47 am PST) by patrickm

We apologize for the downtime with the pyongyang server at this time.  We are working to resolve the problem and will update this post when we have more information and when the server is back up and running!

Update 3pm PST – This issue has been resolved for the last couple of hours. There was a DDoS attack aimed at the server and it drove up the load. We were able to divert the attack and get things back to normal and things on this server seem to have been stable since then. If you still have issues, please contact support.


12:10 pm

seabiscuit going down at 5pm; naroga, petrelli at 5:20p

Posted (April 23rd, 2010 at 12:10 pm PST) by douglas

naroga, petrelli, and seabiscuit were getting a little jealous of all the other MySQL servers’ getting RAM upgrades, so they’ll be getting upgrades too! First to go down for surgery will be seabiscuit at 5pm, followed by naroga and petrelli at 5:20pm.

Update: All upgrades went well, and the mysql servers are now happy. Seabiscuit got a little scared though, it thought that it was on it’s way to the glue factory haha! Not today…


8:00 pm

Easy one-click installs going down Thursday!

Posted (April 22nd, 2010 at 8:00 pm PST) by douglas

The easy one-click install database server will be going down on Thursday, April 22nd at approximately 8pm pacific time, for about 10 minutes. This is for a memory upgrade to boost it’s ram capacity by a small factor of four.

This will only affect DHApps customers, not regular shared hosting or private servers, so none of you paying guys will be affected by this.

Update 8:28PM PDT: RAM upgrade went as planned without any hiccups, so all DHApps services should be restored and in working order. If you’re experiencing any issues, please contact technical support!


5:57 pm

fwap and masevo going down for RAM upgrade at 6:15p

Posted (April 22nd, 2010 at 5:57 pm PST) by douglas

And more to go!

fwap and masevo MySQL servers are also going to get their RAM doubled at 6:15pm pacific time.

Update: All done!


5:07 pm

bennett and cykotix going down for RAM upgrades at 5:20p

Posted (April 22nd, 2010 at 5:07 pm PST) by douglas

Well well, today is the day for MySQL server RAM upgrades.

benett and cykotix will be going down for RAM upgrades shortly. These go pretty fast, so expect minimal downtime.

Update: All done! Now for the others..


12:34 am

Emergency Network Maintenance at our LAX Datacenter

Posted (April 21st, 2010 at 12:34 am PST) by patrickm

We’re very sorry about the inconvenience this is causing, but we are experiencing some network issues with some servers in our LAX data center.  This is causing downtime for Private Servers and MySQL.  Our network engineer is on site and we hope to have this issue resolved as quickly as possible.  Please check this space for updates.

Update: This was fixed the same day that it happened.


 
© 1996-2010, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
11:00 pm

Network Maintenance Window: 11 pm April 18th – 3 am April 19th, 2010

Posted (April 18th, 2010 at 11:00 pm PST) by robertn

There will be a network maintenance window this Sunday night/Monday morning.  This maintenance WILL involve some brief (5 -10 minutes) disruption in service, as we reboot several of our customer-facing switches.  I will post a list of servers that will be involved later this week.

And here is rack k9:

budapest, newdelhi, homie-vserver126, conakry, guatemalacity, bissau, athens, banjul, accra, port-au-prince, tegucigalpa, berlin, saintgeorges, georgetown, tbilisi, reykjavik, homie-vserver124, homie-vserver125

Rack K9 has been completed…Thank you for your patience.

And rack x7:

randymaildb1, lead, sarabi, elmyra, tammany, tachibana, marbles, nanook, molly, clancy, arnie, ned, zorg, salt, manjula, godfather, drederick, jasper, mccallister

And rack X7 has been completed.  This ends tonight’s maintenance window.  Thank you for your patience!


1:25 am

The ‘gap’ is going out of style

Posted (April 17th, 2010 at 1:25 am PST) by Sandon

One of our web-servers ‘gap’ has had a hardware failure (it just power-cycles itself in a loop) and thus we need to move it to new hardware. This process can take anywhere from 30 minutes to an hour. More updates to come.

Update 02:25 AM PST: Most of the process has completed and service should be restored within 10 minutes or so.

Update 02:37 AM PST: Ok, all services on gap have been restored. The good news is that it is now on much newer/beefier hardware than it was before so it should be faster than the old machine.

If you are on gap and having problems with any of your websites please contact support.


8:47 am

Earthquake’s a movin’ and a shakin’!

Posted (April 16th, 2010 at 8:47 am PST) by JeffH

Actually, it’s sorta not…The server, Earthquakes, is currently in single user mode (and that user ain’t you!), running a file check.  This may take a while, so please bear with us if you’re on that server.  This is ultimately a good thing, as it will find and repair any problems with the file system, but in the mean time, the server will be unavailable.  This was not a planned event, and we apologize for any inconvenience.  If you’re having issues and you’re not on Earthquakes, please contact support!

Update (18:20 Pacific): Still fsck’ing. Phase 1 has completed, and now it’s well into phase 1B. No the machine hasn’t failed, just has a lot of data to go through. We’ll have it back up tonight ready for the weekend surfer rush!


5:38 am

Customer Web Panel downtime

Posted (April 15th, 2010 at 5:38 am PST) by patrickm

We are currently experiencing an issue with the customer Web Panel which is causing extreme slowness and downtime.  We are working to resolve this issue now and will update this post as soon as we have more information.  We truly apologize for the inconvenience this is causing.


3:45 pm

DreamHost phishing scheme

Posted (April 14th, 2010 at 3:45 pm PST) by danwy

We’ve just been informed by one of our customers that scammers are sending out emails that resemble official DreamHost billing notices. These messages have their headers spoofed, so it appears that it is coming from a dreamhost.com address when it’s really originating from outside of our system. The example that was brought to our attention was a payment due notice saying that there was a balance due on the account. While the email looked official (down to the ‘from’ address), the links in the email were pointing to a phishing site that was likely designed to steal your panel login information. If you have any reason to believe that you have been a victim of this scam, you should login and change your panel login information immediately. This can be done by clicking the ‘edit profile’ link at the top right hand corner of your web panel and going to the ‘Security’ section.

One thing you can check to verify that the message came from us is to view the full headers of the message. If you read them from the bottom up, the first (bottom most) ‘Received from’ entry will always show as originating from a dreamhost.com server. If there’s any question as to whether or not you’re being directed to the correct address, you should always go directly to https://panel.dreamhost.com to login instead of clicking a link found in an email.


9:28 am

Tarpon on the line (MySQL)

Posted (April 14th, 2010 at 9:28 am PST) by JeffH

The MySQL server, Tarpon is currently experiencing some disk errors which are preventing it from functioning!  We apologize for any issues this may be causing anyone.  If you are on the Tarpon MySQL, check back here for status updates.   If you are not, send us a support ticket!

— Update —

Sorry for the delay, the server is back up and working. After the first reboot and fsck disk check, it came up normally but had issues again. We replaced the raid card with a new one, and it is back up now and we are watching it closely. Our apologies for the problems.


2:18 pm

Git upgrades

Posted (April 13th, 2010 at 2:18 pm PST) by AndrewF

For all of you crazy people using Git, version 1.7.0.4 has just been installed on a bunch of our servers, and should be available on all servers in the Homie cluster by the end of the day. (You are probably in the Homie cluster if you signed up after June 2008, or if you’ve been moved to a new server in the last two years or so.)

If you are not using Git to manage your source code and/or other files, this upgrade will not have any effect on you whatsoever — Git is not a component of any packages we currently provide.

If you are using Git, have fun. :)


1:28 pm

Webmail Downtime 4/14 @ 8:00PM PST

Posted (April 13th, 2010 at 1:28 pm PST) by jordan

Tomorrow night starting at 8:00PM PST we will be upgrading our webmail service. This upgrade will include some database maintenance as well as a version bump from 1.4.19 to 1.4.20 for squirrelmail. The outage should last about 2 hours at most but we should be back up and running by 9:00PM PST. Following this upgrade of squirrelmail, roundcube will also be getting an upgrade in the new few weeks to the stable version 0.3.1.

10:28 PM PST Webmail service has been restored. If you experience any problems please contact technical support.


10:10 am

PHP mail relay clog

Posted (April 13th, 2010 at 10:10 am PST) by andrea

Due to a spam run on a few of our web servers, the php mail relay machine is clogged up a bit. We’re working on cleaning it up, but in the meantime, you may find it slow or impossible to send mail from your webserver. We apologize for the inconvenience, and will post again when things are looking better.

UPDATE 1:30pm Pacific Time: Noting the spam-affected web servers: flames, port-au-prince, huntress, sunpot, burke, twoface and mets.
Some of these machines have a hard time sending still because of their large mail queue, and the relay machine is still clogged up.

This problem affects email sending through web servers only, no regular mail accounts or mail receiving are affected. There shouldn’t be any mail loss either, except for the mail we deem spam and delete to clear up the clog. No estimate on when it’s going to be fixed, but rest assured that it’s being looked into. We’ll post again when there is more news.

UPDATE 4:07pm Pacific Time: All affected machines’ mail queues are now clear, and the relay server is passing mail through again. Sorry about the delay, and if you still need help with your php mail, please contact support with the details through the panel at https://panel.dreamhost.com/index.cgi?tree=support.msg&

UPDATE: 10:08am – 04/13/2010 – One of our mail machines is definitely still getting hammered by what looks to be a big spam run.  We’re working to clear things out right now.  Again, this may cause difficulty in sending mail.  We apologize for the hassle, and we’ll get things cleared up ASAP.

UPDATE: 1:18pm – 04/13/2010 – We’ve one again cleared out the mail queue that was causing these mail delay issues and things should be back to normal. We’ve narrowed down this issue to a handful of customers FTP accounts (77 to be exact) having their password compromised. Based on our server logs, it looks like the IP’s that were responsible for this tried accessing hundreds of FTP accounts on our system but were only able to get into 77 of them. This attack was clearly from a botnet that was using stolen FTP credentials obtained via malware (or some other nefarious method) and not due to lack of security on our end. We have changed the passwords on these affected accounts and have disabled the domains that were being used to send the mail. Our abuse team will be sending out an email to the affected accounts shortly with the new FTP password and further information on how to make sure your site is secure.


3:08 pm

Baku experiencing some difficulties

Posted (April 9th, 2010 at 3:08 pm PST) by Ralph

We are currently working on the hosting machine Baku. It was having problems and we could not get into the machine to see what was wrong. This required us to shutdown the machine. It is currently having problems coming up again but be assured we are actively working on it.

Update: 3:46 PST Baku had a raid failure that required us to replace it. Once replaced the machine came back without any issues.


 
© 1996-2010, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
12:34 am

Highlanders got their head chopped off (RESOLVED)

Posted (April 8th, 2010 at 12:34 am PST) by Sandon

Ok, not really but the server was throwing some file-system errors and remounted back into read-only mode so I had to take the machine down to run a fsck (file-system check). This process can take anywhere between 2-5 hours depending on the amount of data on the disk. More updates to come as they are available.

UPDATE 05:30 Apr 08: filesystem checks are completed and highlander’s head has returned.

If you still experience problems, please contact technical support through normal channels.

Thank you for your patience while we strive to improve your DreamHost experience.


9:00 pm

Flamenco Broke a Leg

Posted (April 6th, 2010 at 9:00 pm PST) by Leon

The flamenco web server is currently experiencing issues with its NIC, so we’re working on moving it to new hardware. We expect it to be up within an hour but please check back for updates!

Update 10:00PM PDT: Flamenco was moved to new hardware and is now back up and running!


5:14 am

homie-vserver90 filesystem check

Posted (April 6th, 2010 at 5:14 am PST) by Ken

homie-vserver90 is doing an extended filesystem check. Please bear with us while we work on the issue.

Affected ps’s are:

ps14547  ps14571  ps14609  ps14848  ps14864  ps15012  ps15133  ps17590  ps17881  ps20178  ps21537  ps22496  ps22790  ps14427   ps14553  ps14581  ps14616  ps14851  ps14901  ps15019  ps15230  ps17591  ps19590  ps20434  ps21561  ps22507 ps14498  ps14558  ps14585  ps14640  ps14854      ps15004  ps15021  ps15231  ps17641  ps19648  ps20503  ps21583  ps22527 ps1450 ps14563  ps14591  ps14646  ps14860  ps15006  ps15075  ps16894  ps17723  ps19904  ps21260  ps22230  ps22537 ps14528  ps14567  ps14602  ps14651 ps15009  ps15119  ps17503  ps17766  ps20013  ps21350  ps22485  ps22556


2:29 pm

Mysql outage for part of homie cluster

Posted (April 5th, 2010 at 2:29 pm PST) by Justin@DH

One of our mysql racks has lost networking, and our admins are investigating. They are already at the data center and will work to get this fixed up as soon as possible. This outage is affecting customers on the following mysql servers:

fin flipper morris tigger cheshire barbaro seabiscuit cykotix masevo fwap bennett grendel lingling puff ein
Edit: buck is not affected, which happens to be in the same rack, and on the same switch.

and the host machines for the following MySQL PS servers:

psmysql10766 psmysql11021 psmysql11060 psmysql11263 psmysql11770 psmysql18028 psmysql18041 psmysql18042 psmysql18043 psmysql18094 psmysql18116 psmysql18149 psmysql18150 psmysql3334 psmysql3336 psmysql3338 psmysql3340 psmysql3343 psmysql3345 psmysql3346 psmysql3347 psmysql3351 psmysql3354 psmysql3366 psmysql3680 psmysql3681 psmysql3693 psmysql3696 psmysql3732 psmysql3734 psmysql3738 psmysql3771 psmysql3803 psmysql3804 psmysql3810 psmysql3814 psmysql3830 psmysql5156 psmysql5190 psmysql5202 psmysql5223 psmysql5226 psmysql5239 psmysql5270 psmysql6714 psmysql6718 psmysql6732 psmysql6740 psmysql6795 psmysql6918 psmysql6942 psmysql6945 psmysql7039 psmysql7242 psmysql7246 psmysql7252 psmysql7267 psmysql7284 psmysql7295 psmysql8173 psmysql8179 psmysql8209 psmysql8222 psmysql8295 psmysql8301 psmysql8339 psmysql8367 psmysql8393 psmysql8447 psmysql8757 psmysql8766 psmysql8806 psmysql8807 psmysql8810 psmysql8833 psmysql8904 psmysql8964 psmysql10018 psmysql10026 psmysql10052 psmysql10093 psmysql10135 psmysql10146 psmysql10171 psmysql10172 psmysql10229 psmysql10234 psmysql10252 psmysql10287 psmysql10317 psmysql10351 psmysql10363 psmysql10421 psmysql10428 psmysql10445 psmysql10583 psmysql10767 psmysql10837 psmysql10849 psmysql11160 psmysql11545 psmysql11888 psmysql11954 psmysql12022 psmysql12076 psmysql13691 psmysql13835 psmysql13849 psmysql13971 psmysql14270 psmysql14632 psmysql16076 psmysql16367 psmysql18019 psmysql18029 psmysql18057 psmysql18077 psmysql18151 psmysql18172 psmysql19104 psmysql19167 psmysql19168 psmysql19170 psmysql19172 psmysql19173 psmysql19175 psmysql19176 psmysql22296 psmysql22591 psmysql9550 psmysql9553 psmysql9557 psmysql9559 psmysql9560 psmysql9596 psmysql9599 psmysql9600 psmysql9603 psmysql9604 psmysql9605 psmysql9671 psmysql9672 psmysql9677 psmysql9678 psmysql9679 psmysql9680 psmysql9682 psmysql9684 psmysql9686 psmysql9688 psmysql9689 psmysql9691 psmysql9692 psmysql9694 psmysql9709 psmysql9712 psmysql9718 psmysql9783 psmysql9802 psmysql9848 psmysql9862 psmysql9894 psmysql9971 psmysql9975 psmysql9979

We will post more info when we have an ETA or the issue fixed. Our apologies for this unexpected outage.

UPDATE: All fixed! Our network guy will still be watching the cores for any other unusual activity.


2:30 pm

homie-vsql17 bad RAID

Posted (April 2nd, 2010 at 2:30 pm PST) by douglas

So, homie-vsql17 went down earlier today due to a bad raid battery. After restarting it turns out it was more than that, in fact the RAID card was bad. So it’s down again. After the raid card replacement, restart, and subsequent fsck, it’ll be back online. This system uses very high speed 2.5″ SAS drives, so fsck will run quickly on this machine.

Update: New raid card has been installed, and all vsql guests appear up.


11:22 am

Barkley is a terrible role model for both kids and MySQL servers

Posted (April 1st, 2010 at 11:22 am PST) by JeffH

Our apologies!  It seems our MySQL server, Barkley, has come to irreconcilable differences with its power supply.  We’ve got an admin working on replacing it, and should have it back up within an hour or so.  It’s also being sanctioned by the NBA for it’s behavior.

If you’re not on Barkley, your problems are not related.  These are not the droids you’re looking for…move along (to the support panel, and file a ticket).

Update: This is now resolved as the machine is back up and running fine. We apologize that it took a bit longer than anticipated due to an issue with its Ethernet ports which was not caught immediately but is now corrected.


11:51 pm

homie-vserver17 hardware replacement – RESOLVED

Posted (March 31st, 2010 at 11:51 pm PST) by patrickm

Due to this host server crashing a lot lately, we’re moving it to new hardware, which should help resolve that issue.  This will cause intermittent downtime for the following Private Servers while we complete the maintenance:

ps10345 ps10584 ps10877 ps11175 ps11575 ps11985 ps12189 ps13213 ps14768 ps14811 ps14815 ps14950 ps14978 ps15208 ps15210 ps15255 ps15257 ps15258 ps20680 ps21008 ps21217 ps21301 ps21377 ps21816 ps22011 ps22329 ps22625 ps23001 ps23920 ps23962 ps5441 ps5464 ps5467 ps5473 ps5482 ps5502 ps5505 ps5659 ps5663 ps5669 ps5672 ps5778 ps5817 ps5820 ps5837 ps5885 ps5899 ps5907 ps5979 ps6154 ps6156 ps6198 ps6208 ps6218 ps6226 ps6396 ps6412 ps6471 ps6473 ps6481 ps6487 ps6695 ps6704 ps6750 ps8089

We apologize for the downtime and will update this post once we have completed this change!

Update 1:10 am Pacific: Thanks for your patience!  This maintenance has been completed and the host’s services have been restored.  If your PS is listed above and you’re still having any issues, please contact Support for assistance!


3:07 pm

Brief network flap

Posted (March 30th, 2010 at 3:07 pm PST) by danwy

We had a brief network flap caused by one of our vlans running out of IP’s and resulted in downtime for about 10 minutes on a handful of our servers. This issue has been resolved and everything that was affected is back up and running. If you are still experiencing downtime, it is not related to this issue and you should contact support to investigate it further.


12:18 am

panamacity fs errors

Posted (March 30th, 2010 at 12:18 am PST) by Sandon

It looks like the web-server ‘panamacity’ was getting some file-system errors so we had to take it down to do a fsck (file-system check). This can take anywhere from 2-5 hours depending on the amount of data on the file-server. More updates to come as they are available.

Update (2:37 AM PST): Its finished several of the passes and is likely down to less than 30 minutes done on the FSCK.

UPDATE 30 Mar 04:40: After some extra checks and tests, panamacity is now back up. Thank you for your patience while we strive to improve your Dreamhost experience.

UPDATE 31 Mar 01:50: The machine is having file-system errors again and we believe the problem has been traced to a bad SATA cable. An fsck has to be re-ran which means the machine will likely be down another 3 hours with the FSCK. If we have any additional problems with the machine we will likely be restoring it from backups on new hardware.

Update 31 Mar 4:36: These issues have been resolved for the last few hours. We wanted to monitor the situation a little longer to make sure things remained stable. We’ve tested most of the sites that were affected and have verified that the backups have finished restoring. The FTP permissions errors and the site downtime should be resolved now. If you have any further problems, please contact support.


10:44 pm

Network Maintenance window Sunday 3/28-29/2010 11pm -2 am

Posted (March 28th, 2010 at 10:44 pm PST) by robertn

3/28/2010  Starting now 11:07 pm

3/29/2010 3:52 am  Maintenance window is closed.

There will be a maintenance window for some non-intrusive/non-disruptive maintenance on our network infrastructure this Sunday night.  As your Happy Dreamhost Staff strives to bring you the highest quality in hosting, we will be:

1.  Improving our Network Time Protocol support

2.  Standardizing our network infrastructure at our LAX data center.

These improvements should not cause any disruption in service.  As always, if you notice any problems, please call our technical support staff.


 
© 1996-2010, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
12:35 pm

homie-vserver02

Posted (March 26th, 2010 at 12:35 pm PST) by douglas

Users on homie-vserver02 will be experiencing momentary outages as we adjust the way memory is handled on the vserver. This is to address some performance issues we’ve been encountering with various PS’.

Affected PS':

bcboo			ldsseminary		ps12921			pyrite
cardamom		liupoyuan		ps12963			qubitron
caucus1			lunari			ps13138			rbk
clzvps			nbacharters		ps13184			rockingromans
creativesource		nevin			ps15476			shamble
csiribirifostalyiga	pastelserver		ps15582			stevemccurry
cybercomuk		pbbnz			ps15679			tastespotting
cynsspot		peregrine		ps15834			todesengel
eglobal			pixellab		ps22524			tyax
gauss			proxy1			ps22529			urbansonar
golfzone		ps11106			ps22531			vegan
goog1			ps12206			ps4164			voicer01
indev			ps12911			ps4171

Update: The configuration changes are complete. Your PS will be restored shortly, if not already.


10:27 am

Private Server Downtime

Posted (March 26th, 2010 at 10:27 am PST) by Kyle

A private server host went down this morning and has to finish a filesystem integrity check before it returns to service. We’re sorry for the inconvenience! Guests effected:

ps10186 ps10189 ps10379 ps14131 ps14188 ps14354 ps14494 ps15127 ps15147 ps15148 ps18484 ps18485 ps18486 ps18487 ps20426 ps20499 ps20554 ps20955 ps21185 ps4633 ps4638 ps4651 ps4653 ps4666 ps4675 ps4810 ps4817 ps4842 ps4855 ps4863 ps4868 ps4898 ps4904 ps4914 ps4918 ps4921 ps4931 ps5018 ps5036 ps5042 ps5043 ps5050 ps5053 ps5056 ps5063 ps5068 ps5071 ps5075 ps5089 ps5098 ps5102 ps5103 ps5619 ps5623 ps5765 ps5770 ps5974 ps6144 ps6160 ps9841

Update 10:38 PST: The filesystem check has finished and guest services have been restored!


10:35 am

Web server “elise” is under the weather.

Posted (March 25th, 2010 at 10:35 am PST) by David

Unfortunately one of our web servers “elise” is down and needs special attention. Technicians have already been dispatched to investigate and get it back on-line again. Please check back here for updates.

The Happy Server-Rescue Team

Update: 10:45am PDT: The server had to be power-cycled but after doing so it has rebooted successfully and seems to be working fine now. If you find any problems with your sites (hosted on this server) please submit a support request through the normal channels.


3:46 pm

Karpis on the move, but not yet busted!

Posted (March 23rd, 2010 at 3:46 pm PST) by JeffH

10 points to whoever got the reference!

http://en.wikipedia.org/wiki/Alvin_Karpis

Karpis (the server, not the outlaw), is having accounts moved off of it to improve stability.   Unfortunately (and ironically) the moves themselves are making the machine work pretty hard.  The upside is that things will be all around happier when they are done!   We appreciate your patience if you’re on this machine.  We’ll post an update when it’s done!

If you’re not on Karpis, this doesn’t affect you, so you should contact support if you’re having issues!   But be nice…they are a hard-working bunch of folks!


9:59 am

Webmail problems

Posted (March 22nd, 2010 at 9:59 am PST) by danwy

We’re currently experiencing some issues with our webmail interface. Our admin team is aware of the issue and is working on resolving it now. We don’t hve any details at this time, but will update this post as soon as any new information is available.

Update – 11:15am (PST): This issue has been resolved for the last 30 minutes or so as far as we can tell. It looks like the load balancer was having some issues and the loads on the webmail machines rose significantly because of it. Everything is back to normal now and all of the mailboxes that we’ve tested in webmail are functioning correctly now. If you have any further problems, please contact support.


8:25 am

Esprit hardware issues, new server being setup

Posted (March 21st, 2010 at 8:25 am PST) by Justin@DH

The webserver esprit is having problems booting. One of our admins is on the way to setup a machine to replace it and get it back up. We’ll be working to get this resolved as soon as possible, and update this post when we have more info.

Update: The server was setup on new hardware and is back up. Sorry for the problems this morning.


3:43 am

Penguins, Match Penalty

Posted (March 20th, 2010 at 3:43 am PST) by Kyle

Penguins was in the process of undergoing a scheduled kernel upgrade and when rebooted to load the new kernel it refused to power back up.  Remote hands have identified it as a RAID issue so we are dispatching a admin to the datacenter to coax it back up.

Update 04:30 PST: On site and working on getting penguins back up.

Update 04:42 PST: We were able to sweet talk the server into behaving and services are coming back online right now.  Please contact support if you are still experiencing problems.


12:00 am

Kernel Upgrade March 12 – Midnight PST

Posted (March 20th, 2010 at 12:00 am PST) by Kyle

We are upgrading kernels on the homie cluster at midnight this Friday.  This move will migrate us to the 2.6.32 kernel series which is the next long-term stable branch that is being supported by the stable team (replacing 2.6.27).  Each machine will be brought offline only for the amount of time required for a reboot so downtime will be minimal for individual sites.

March 13 00:27 PST: I updated the post date to now so that it would show up at the top of the status page.  We are beginning with servers starting with “A” now.

March 13 01:05 PST: “A” machines are finished starting on ones beginning with “B”.

March 13 02:10 PST: “B” machines are finished starting on ones beginning with “C”.

March 13 02:42 PST: “C” machines are finished starting on ones beginning with “D-F”.

March 13 04:00 PST: “D-F” machines are finished starting on ones beginning with “G-H”.

Remaining servers will be scheduled for a later date.

March 17: We are planning to do I-Z this Friday at midnight, just a heads up!

March 20 00:10 PST: It’s kernel time! Starting with servers beginning with the letters I-L.

March 20 1:00 PST: M-N time!

March 20 2:00 PST: O-P time!

March 20 2:40 PST: Q-R time!

March 20 4:47 PST: S time!


5:50 am

Maputo Mapputatsu! – RESOLVED

Posted (March 19th, 2010 at 5:50 am PST) by Sandon

The web-server ‘Maputo’ has had a hardware failure causing the machine to power on but not have any type of VGA output or any activity. Because of this someone has been dispatched to the Data Center to move the drives to shiny new hardware to get the machine back up. Assuming no problems the ETA on this is about 30-40 minutes. More updates to come!

UPDATE: sorry – that was MAPUTO not MAPUTA.

UPDATE MAR19 @ 08:11: Server maputo is now back up on brandy new hardware.

If you continue to experience problems, please contact technical support through the normal channels.

Thank you for your patience while we strive to improve your DreamHost experience.


9:12 pm

Tick filesystem troubles – RESOLVED

Posted (March 18th, 2010 at 9:12 pm PST) by DreamHost

Tick is experiencing some problems with one of it’s filesystems, and is currently offline for maintenance. We don’t yet have an ETA when it’ll be back online, but likely several hours. We’ll update when we have more info.

UPDATE MAR 19 @08:04: tick is now ticking and services restored.

If you continue to have issues with your services and you’re on tick, please use normal channels to report them.

Sorry for the inconvenience and thank you for your patience while we strive to improve your experience.


 
© 1996-2010, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
3:32 pm

Zorg Out of its Element

Posted (March 18th, 2010 at 3:32 pm PST) by Leon

The zorg web server is currently experiencing hardware issues and we’re working on moving it to new hardware. We expect it to be back up within the next hour or so but please check back for updates!

Update 6:00PM PDT: Zorg is now back on its new hardware and should be chugging along without any issues. If you’re still experiencing issues and you’re on the zorg server, please contact our support staff!


2:06 pm

humantorch down

Posted (March 17th, 2010 at 2:06 pm PST) by douglas

humantorch is suffering form some filesystem trouble after the RAID backup battery died. Running filesystem checks to verify and fix filesystem inconsistencies before bringing the system back online.

There is no ETA for the moment, but expect hours.

Update: We replaced the RAID card and found that the /home partition was fine, but the root partition (os) was not. We’re now re-imaging the machine and should have it back around an hour.

Update 03/17 10:20 PM (PST) This machine was back up and running for a couple hours but now the customer data partition is throwing some file-system errors and went back to read-only mode so we have to take the system down to run an FSCK (file-system check). Now that it has a new raid controller it should be good once the FSCK finishes. Unfortunately this can take anywhere from 1 to 5 hours.

Update 03/18 9:40 (PST): It’s back up and rocking, if you are still experiencing problems please contact support.


12:30 pm

Hoffa Not All There

Posted (March 16th, 2010 at 12:30 pm PST) by brettg

Our web server hoffa has been experiencing some load issues recently; this was caused by a drive failing in the Raid array, which we then promptly swapped out. The drive is currently rebuilding which will maintain the current I/O performance until the the drive is fully rebuilt. Once the drive is finished, I/O performance will continue to operate under normal provisions. We will update you once this has finished, if you have any questions please contact our support department!

The Array-Rebuilding-Dreamhost Team

Update 03/17/10 10:40 PM PDT The array has almost finished rebuilding and in the meantime the I/O scheduler has been changed quite some time ago which helped bring down the load and iowait on the machine so it isn’t experiencing excessive load averages or slow I/O anymore.


4:37 am

Earthquake in Los Angeles

Posted (March 16th, 2010 at 4:37 am PST) by patrickm

This morning, Los Angeles county experienced a minor earthquake that was centered about 20 miles from our Downtown Los Angeles data center (4.4 magnitude in Pico Rivera).  We wanted to update everyone that our engineers have thoroughly checked our systems and we suffered no damage, nor downtime, due to this event.  Just a little shakin’ around!

http://earthquake.usgs.gov/earthquakes/dyfi/events/ci/14601172/us/index.html

–The Happy DreamHost Earthquake Response Team


9:49 pm

Sabretooth Part Deux – RESOLVED

Posted (March 15th, 2010 at 9:49 pm PST) by Ken

Apparently, not only did sabretooth raid go offline again, there was a bad drive as well. The bad drive has been replaced. It’s currently going through another drive check and should b done around Mar 16th at 01:00.

If it continues to have errors, we will be looking at replacing the raid card as well, but that will be looked at after this check is done.

UPDATE 16 Mar 02:20: Still doing drive checks, so it looks like it may take a couple of more hours due to the drive replacement. We thank you for your patience while we work on this.

Update 16 Mar 07:35: This file system check has now been completed!  The server is back online and services have been fully restored.  Please contact support if you are still seeing any related issues!


9:17 pm

Sabretooth lost its teeth

Posted (March 14th, 2010 at 9:17 pm PST) by Ken

Sabretooth is having raid issues. We are currently running an fsck on the drive which can take up to 2-3 hours and will update this post when finished. Until the fsck is up, all services on sabretooth will be down.

We regret the inconvenience and will get services restored as soon as possible. Your Dreamhost NOC team.

UPDATE 2010 Mar 15 @03:59: Sabretooth has started pass 1C in the filesystem check, so it shouldn’t be too much longer now.

UPDATE 2010 Mar 15 #04:37: Sabretooth is now back up. If you experience problems with your service, please contact us through the normal channels. Again, thank you for your patience.


8:01 am

Amman file-system errors

Posted (March 13th, 2010 at 8:01 am PST) by robert

Due to file-system errors, we had to restart the web server amman into single-user mode and run a fsck (file-system check). This process has been known to take anywhere from 1-5 hours depending on the amount of data/files on the server. Unfortunately your sites/data on the server will be inaccessible until the fsck is done. More updates to come as they are available.

Update: 11:50 AM PST: The FSCK has finished. Please contact support if you are still experiencing issues!


10:50 am

Cruisin hardware issues

Posted (March 12th, 2010 at 10:50 am PST) by robert

The web server cruisin is currently experiencing issues with its hardware upon reboot. We’re working on getting it moved onto new hardware.

Update: Cruisin is on new hardware and all services have been restored. If you’re still experiencing issues, please contact support!


12:53 pm

Brief Mail problem on Homie

Posted (March 10th, 2010 at 12:53 pm PST) by danwy

Some of you may have noticed a brief outage on our homie mail cluster. This resulted in an “Access denied” bounceback when people tried to email you. This was due to a misconfiguration on our end and was resolved in about 10 minutes. We’re in the process of running configs to catch any stragglers, but this issue should be resolved at this point. If you’re still receiving “Access Denied” bouncebacks, please let us know and we’ll be happy to investigate it further for you.


11:08 am

DreamHost Apps Downtime

Posted (March 10th, 2010 at 11:08 am PST) by Kyle

We are currently upgrading the distribution release for DreamHost Apps, during this process DreamHost Apps sites will be unavailable.  We are sorry for the short notice!

Clarification: The signup/management pages are being worked, we ran into some unexpected complications and we don’t have a ETA as of yet on restoration of service.


 
© 1996-2010, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
12:53 am

Zatara file-system errors

Posted (March 10th, 2010 at 12:53 am PST) by Sandon

The web-server “Zatara” was getting some file-system errors and thus remounted itself in to RO (read-only) mode. Because of the errors an fsck (file-system check) needs to be ran on it which can take anywhere from 1-5 hours (depending on the amount of data on the machine) in which time customer data and web-sites hosted off the machine will be down. More updates to come as progress is made on the FSCK.

Update: 03:42 AM PST: The first pass has finished which typically manes this is around 50% complete.

Update: 08:05 AM PST: The FSCK has finished and these issues should be resolved now. We’ve tested several sites and verified that everything is back up and running. If your site is on this server and you’re getting a bad_httpd_conf error still, be sure to clear out your browsers cache. If you have any further issues, please contact support.


1:30 pm

Maintenance Window: March 7th, 11 pm

Posted (March 7th, 2010 at 1:30 pm PST) by robertn

Note:  This maintenance was completed successfully.  3/7/2010 11:34 pm

There will be a network maintenance window this Sunday night/Monday morning.  We will be changing routing policies and improving our “blackhole” server to better protect you.  There should not be any disruption in service, but this work does affect our border routers and core routing.  Thus, there is a chance of a network blip or flap.

We strive always to maintain the highest standards of security and service.  If you notice an issue, please let one of our technical service representatives know immediately.


9:09 am

Postal filer problem

Posted (March 6th, 2010 at 9:09 am PST) by robert

One of our filers, cutter, has been experiencing issues this morning. The following machines are affected and users on them can/will experience high load or unavailable data while we work on the issue:

abe amw annie apu arnie barriodelcarmen bart bitters blatant bloodcopy brandine brockman burns carl chalmers cheeky clancy cletus drederick duffman gaz gimli gir glass gloop gnif gotcha harpo hibbert homer hyperblocks jasper jebediah jimbo jubjub kearny krabappel krusty lablogs larry lisa lovejoy manjula mccallister millhouse mita mona monkey mtrends ned nelson osteele otto patty porker ps3845 ps3927 ps4326 ps4473 ps5389 ps6300 ps6428 ps7922 quimby raiden ralphie salt sanjay scooter seamus shemp skinner softwa syscount tak touchofdeathforums vivartes vpg willie yerba yggsdrasil zim

Update: Sorry for the late update, the servers posted above are stable shortly after this post. If you’re still having problems with your domains, please open a ticket with our support team.


1:03 pm

homie-vserver152 down (hardware failure)

Posted (March 4th, 2010 at 1:03 pm PST) by douglas

homie-vserver152, on hardware only a few months old, had it’s power supply die. We are working to restore the system now.

Affected guests:

ps12554    ps21840    ps6906    ps7325    ps7495    ps8144
ps19800    ps21859    ps6914    ps7329    ps7732    ps8236
ps20081    ps22467    ps6933    ps7342    ps7754    ps8240
ps20261    ps22492    ps6954    ps7373    ps7757    ps8320
ps20278    ps22500    ps7003    ps7378    ps7759    ps8321
ps20538    ps6870    ps7022    ps7411    ps7853    ps8333
ps21697    ps6874    ps7030    ps7436    ps7975
ps21790    ps6894    ps7050    ps7447    ps8021
ps21815    ps6898    ps7201    ps7493    ps8142

Update: We lied. it’s the motherboard not the power supply… we’re getting new hardware prepared for the drives, we may also need to run a filesystem check to before bringing up guests.

Update: We’ve had homie-vserver152 back up for awhile.


6:01 pm

Homie-vserver142

Posted (March 3rd, 2010 at 6:01 pm PST) by Leon

Oops! Looks like this machine wasn’t setup quite correctly and is starting to have some issues. We’re re-creating it and moving your data araound to the correct spots. Should be working again within an hour or so! Affected PSes are:

ps22487 ps22530 ps22533 ps22535 ps22538 ps22539 ps22540 ps22541 ps22542 ps22543 ps22544 ps22545 ps22546

UPDATE Mar 04, 03:48PST: After finally getting a good copy back onto homie-vserver142 and restoring customer data, homie-vserver142 has returned.

If you are still having issues with your personal server, please contact Tech Support via normal channels.


12:14 pm

Canceled: Scheduled Power Outage 3/6/2010 — Garland Building

Posted (March 3rd, 2010 at 12:14 pm PST) by robertn

Canceled:  No work this weekend.

We have been informed of a scheduled maintenance window by the building management company for this Saturday, March 6th, 2010.  From their email:

Start time: 6:00am pacific 03/06/10
End time: 12:00pm pacific 03/06/10
Expected Duration of Power Outage: 6 hours

This maintenance has been scheduled by Morlin Management  who should have already notified you. The purpose of this work is building power maintenance.

Our data center does have backup power generators, and this should not cause a disruption of service.  We strive to keep our customers informed of ongoing improvements.  This shouldn’t hurt a bit…


12:01 pm

Alchemy Scheduled Maintenance — LAX

Posted (March 3rd, 2010 at 12:01 pm PST) by robertn

See?  Now that didn’t hurt a bit!  Maintenance completed successfully.

Our friends at the LAX Alchemy, where we have one of our Data Centers, has informed us they will be doing maintenance on their UPS.  From their email:

A preventive maintenance on some of Alchemy’s UPS power at the LAX data center will be performed during a scheduled maintenance window beginning on Wednesday, March 3rd, 2010, at 10:00pm (data center local time) and ending approximately six (6) to eight (8) hours later (2200-0600). As a precaution, all power will be shifted to generator power during this maintenance. The maintenance should be transparent to you and no interruptions of service are anticipated; however as with any maintenance, there is a small possibility of service issues as configuration changes are made.

This notice is just to keep you informed!  You shouldn’t feel a thing…


10:00 pm

Panel Database Maintenance, February 26, 10PM PST

Posted (February 26th, 2010 at 10:00 pm PST) by DreamHost

We will be performing maintenance on our central database later this week, beginning Friday, February 26 10:00PM PST, ending 11:59PM PST. This will affect the following services:

DreamHost Webpanel
DreamHost-provided Webmail
Signups
Domain Registrations
DreamHost Wiki

Update Fri Feb 26 23:47:32 PST 2010: Everything is back online.


8:15 am

Outgoing mail originating from webserver is being delayed

Posted (February 23rd, 2010 at 8:15 am PST) by brians

There is a mail queue on the machine that handles mail delivery for the webserver. Therefor, any message generated by your website via sendmail, php mail() will be delayed. Our administration team is currently investigating the issue. We sincerely apologize for the inconvenience this has caused an appreciate your continued patience. Please check back for future updates.

UPDATE: There was heavy volume of spam on the machine. We are in the process of removing the spam. This can a little time. Please continue to bear with us. Again, we apologize for the inconvenience this has caused.

UPDATE: As of 12:45pm PST, the mail relay is turned back on , and mail sending has caught up. We continue filtering mail as it comes in, so it may cause minor delays in messages sent from a web server, but otherwise we consider this case closed. Sorry again about the problems.

UPDATE 3:40pm PST: We are now moving back earlier messages from before the fix for delivery. They have been filtered through and examined for potential spam. If you haven’t seen your morning emails from your server yet, they should be arriving soon. Sorry about the further delay.


9:57 pm

Edsel hardware issues

Posted (February 20th, 2010 at 9:57 pm PST) by DreamHost

Edsel has died, and we’re in the process of moving it to new hardware. Sorry folks.

Update Sun Feb 21 00:34:08 PST 2010: Edsel back online as of a bit ago.


 
© 1996-2010, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
5:27 pm

Network Maintenance – Sunday February 28st 11 pm

Posted (February 19th, 2010 at 5:27 pm PST) by robertn

February 28th, 11:00 — this maintenance has commenced.

Brief network flap between LAX and Garland data center as network configs converged.

Completed!

February 21, 19:11  This maintenance will not take place tonight.  It will be rescheduled for next week.

There will be routine network maintenance performed Sunday, February 21st 11 pm until Monday February 22nd 2 am.  The primary focus of this maintenance is config clean up and performance enhancements.

There should be no customer impacting procedures.  This post is purely informational and our attempt to keep you – our valued customer, informed.  If you do notice any anomalies in performance, please contact your Happy Dreamhost Customer Support representative immediately.


6:00 am

homie-vsql12 getting more ram

Posted (February 18th, 2010 at 6:00 am PST) by robert

homie-vsql12 will be going down for ram upgrade at about 6:15am PST today. Will be back up within 30 minutes. Affected MySQL PSes are:

psmysql10016 psmysql10020 psmysql10050 psmysql10062 psmysql10138 psmysql10143 psmysql10148 psmysql10153 psmysql10159 psmysql10177 psmysql10236 psmysql10245 psmysql10254 psmysql10289 psmysql10316 psmysql10394 psmysql10402 psmysql10426 psmysql10442 psmysql10449 psmysql10580 psmysql10622 psmysql10649 psmysql10703 psmysql10713 psmysql10751 psmysql10763 psmysql10834 psmysql10920 psmysql11015 psmysql11264 psmysql11560 psmysql13672 psmysql13685 psmysql13735 psmysql13944 psmysql13963 psmysql14156 psmysql14467 psmysql14504 psmysql14506 psmysql15446 psmysql16184 psmysql16208 psmysql16354 psmysql16610 psmysql18050 psmysql18068 psmysql18069 psmysql18107 psmysql18135 psmysql18162 psmysql18163 psmysql19056 psmysql19110 psmysql19247 psmysql9051 psmysql9056 psmysql9178 psmysql9184 psmysql9190 psmysql9192 psmysql9194 psmysql9198 psmysql9200 psmysql9202 psmysql9204 psmysql9208 psmysql9210 psmysql9216 psmysql9218 psmysql9220 psmysql9222 psmysql9224 psmysql9228 psmysql9230 psmysql9234 psmysql9251 psmysql9255 psmysql9348 psmysql9350 psmysql9352 psmysql9356 psmysql9359 psmysql9365 psmysql9368 psmysql9371 psmysql9374 psmysql9713 psmysql9778 psmysql9780 psmysql9786 psmysql9792 psmysql9796 psmysql9800 psmysql9810 psmysql9815 psmysql9846 psmysql9886 psmysql9902 psmysql9973 psmysql9977

Update: Ram upgrade completed.


12:23 am

ironman (serf2877) is running a disk check

Posted (February 18th, 2010 at 12:23 am PST) by Ken

ironman is currently running through an extended disk check and should be done in about an hour (around 01:30 PST).

We thank you for your patience while these checks are progressing.

UPDATE 00:44 PST: Disk checks finished sooner than expeced. irnoman is back up and services restored. If you are still experiencing problems, please contact tech support via normal channels.

We thank you for your patience while we continue to improve your Dreamhost experience.


12:38 pm

Homie-vserver30 migration

Posted (February 17th, 2010 at 12:38 pm PST) by robert

Due to ECC error on one of the drives, we are currently migrating the private servers on homie-vserver30 to a new host machine.  The process will take several days as I’m moving the private servers one by one during my scheduled work shift (6AM – 2 PM PST) . Affected private servers are:

ps12554 ps19800 ps20081 ps20261 ps20278 ps20538 ps20603 ps20650 ps20959 ps21188 ps6870 ps6874 ps6894 ps6898 ps6906 ps6908 ps6914 ps6933 ps6954 ps7003 ps7022 ps7030 ps7050 ps7057 ps7143 ps7180 ps7201 ps7325 ps7329 ps7342 ps7373 ps7378 ps7411 ps7436 ps7447 ps7493 ps7495 ps7732 ps7754 ps7757 ps7759 ps7853 ps7975 ps8021 ps8142 ps8144 ps8236 ps8240 ps8320 ps8321 ps8333 ps8342

Down time for individual private servers will only be around couple minutes (at most) when the IP switches over to the new host machine.

Update Feb 17 2:30PM PST:  ps12554, ps19800, ps20081, ps20261, ps20278 were migrated today.

Update Feb 21: ps20538, ps6870, ps6874, ps7201, ps7411, ps6894, ps6898, ps6906, ps7003, ps7975, ps6954, ps7329, ps6914 and ps7447 were migrated over the past few days.

Update Mar 03: All done!


12:37 pm

Networking Issues

Posted (February 16th, 2010 at 12:37 pm PST) by Justin@DH

Starting at noon and lasting around 20 minutes, we experienced a network outage that effected some but not all of webservers, mysql servers and PS servers in the homie cluster. Most services are back up and our network admins are looking at the last few issues that are still remaining. As soon as we have further info about the issue we will post an update.

— UPDATE —

The admins have determined that a network connection between our core router and a specific set of server racks was down. This overloaded other parts of the network which caused some slowness. The connection is back up and changes were made to help prevent such a large downtime should it have issues again, while we investigate and eliminate the root cause. Our monitoring system shows that all services have recovered, and we will keep an eye out for any other issues. If you notice any further problems, it may be a specific issue related to just your sites and not to this networking issue in which case please contact support.


10:40 am

podgorica is sad

Posted (February 16th, 2010 at 10:40 am PST) by Kyle

The shared server “podgorica” is having difficulties powering up after a crash.  We have dispatched a technician to the facility to rectify the issue, he should arrive on site in approximately 30 minutes. Details to follow..

-UPDATE Tue Feb 16 14:26:25 PST 2010 –

podgorica has been back up and running for a little over an hour. It’s hardware died and the drives were moved to a new machine. We apologize for any inconvenience and if you have any questions or concerns, please contact support.


9:38 am

Server Migrations (Round 5)

Posted (February 13th, 2010 at 9:38 am PST) by Patrick

We’re continuing the migrations described here. The following servers will be rebooted some time after 6PM PST tonight:

rockets rockies rogue roseau roughriders royals sabretooth sandman sanjose santiago santodomingo sarajevo seahawks shadow shock silversurfer snowbird sofia sox sparks spectre spiderman spurs starfire steelers stjohns sugeknight suns sunspot suva swampthing tallinn terranova texans thing thor thrashers thunder tick tigercats tigers tirane trafficante twins twoface venom vienna walsh waratahs warmachine warriors wasp wizards wolfsbane wonderwoman yankees yaounde yerevan zagreb

The entire process will take a couple of hours and not all machines will go down exactly at six. Down time for individual machines will only be around 5 minutes. Check back here for updates, and as always, if you have any questions or concerns, please contact support!

– Update Sun Feb 14 03:36:19 PST 2010 –

All machines have been converted, and have been up and running for several hours. If you are having any problems whatsoever, please contact support!


3:22 pm

Flash and Files Forever

Posted (February 12th, 2010 at 3:22 pm PST) by AndrewF

One of our internal servers blew a drive this morning, and it’s taking a bit for us to recover the contents. Until we get it back in working order, Files Forever and media.dreamhost.com will be unavailable. This will make Flash videos created by our Flash Media converter unavailable until the machine is back up and running.

Update 4:55 PM PST Friday: media.dreamhost.com is back up, and Files Forever should be up shortly.

Update 5:14 PM: Files Forever is back up now.


2:32 pm

Homie-vserver135 down

Posted (February 11th, 2010 at 2:32 pm PST) by jeremy

Update 8:50pm PST:

We’ve gotten the server back online after replacing the RAID controller and running a lengthy file system check. There is a small possibility that some data may have been lost so if you notice anything missing, please contact support and they should be able to restore your data from the most recent backup.

We’re currently working on fixing some issues with its RAID array so it’ll be down for some time while the problems are corrected.

The following PS guests will be down until the work is completed:

ps19865
ps19878
ps19884
ps19895
ps19911
ps19927
ps19939
ps19953
ps19984
ps20024
ps20140
ps20195
ps20199
ps20221
ps20277
ps20432
ps20483
ps20485
ps20545
ps20612
ps20613
ps20617
ps20653
ps20662
ps20770
ps20798
ps20799
ps20856
ps20929
ps20998
ps21037
ps21108
ps21114
ps21118
ps21246
ps21327
ps21367
ps21397
ps21445
ps21511
ps21649


5:07 am

Possible LAX routing issue

Posted (February 11th, 2010 at 5:07 am PST) by Ken

While monitoring our servers, it was noticed that the servers hosted at our LAX facility started reporting down.

We are actively looking into the issue and appreciate your patience while we work on this.

UPTDATE Feb 11 05:25 PST: There was a temporary issue with the LAX facility that affected all services. All services have been restored. We are continuing to investigate what happened and will update this post when we have more information.

If you are still experiencing service issues, please report them via the normal channels.

UPDATE Feb 12 00:21 PST: After looking some more, it appears that a core router at the Alchemy LAX facility was upgraded with a security update and rebooted. Total outage time was between 5 and 6 minutes.

We regret the inconvenience and appreciate your patience while we strive to improve your DreamHost experience.


 
© 1996-2010, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
5:28 pm

Legs took a hike.

Posted (February 10th, 2010 at 5:28 pm PST) by Leon

Unfortunately, the RAID card in legs went south and we’ve had to replace it. It’s currently rebuilding a drive that was lost in the process and is read-only at the moment.

We will update when we have an ETA as to when it’ll be back up!

Update 5:50pm PST: As mentioned, the raid card was replaced and we needed to bring the server down for a file system check. It’s back up now but we’re going to continue to monitor the server. All services should be in working order as of now.


9:41 am

Server Migrations (Round 4)

Posted (February 10th, 2010 at 9:41 am PST) by Patrick

We’re continuing the migrations described here. The following servers will be rebooted some time after 6PM PST tonight:

marv masseria mclean mekon mets minsk monarchs moroni mothman mrfantastic mystique namor nassau nationals ndjamena nicosia nightwing niners nitti nuggets obelix oilers olympiacos ottawa ouagadougou outlaws pacers packers padres paris patriots penguin penguins phillies phnompenh pirates pistone pistons porto-novo prague praia preacher predators pride professorx quicksilver quito raiders rams rangers rapids raptors ravens rays reds redwings revolution riddler robin

The entire process will take a couple of hours and not all machines will go down exactly at six. Down time for individual machines will only be around 5 minutes. Check back here for updates, and as always, if you have any questions or concerns, please contact support!

– Update Wed Feb 10 22:41:04 PST 2010 –

After a late start, all machines have been migrated and are up and running. Please contact support if you are experiencing problems with your service.


5:09 am

Panel Slowness

Posted (February 10th, 2010 at 5:09 am PST) by Sandon

There is a problem when making changes to domain registrations or checking availability which is taking a very long time to process which causes the requests to eventually pile up and is causing some general slowness of the panel. One of our senior developers is investigating the issue. More updates to come.

Update (2010-02-10: 08:07):

We’ve definitely nailed down the behavior and the specific process that is causing the backup and have identified when the issue started (the process went from taking about 5 seconds to run to 10-15 minutes which is why things are backing up so badly). As soon as we have corrected the matter we’ll update this post again.

Update (2010-02-10: 09:12):

The issue has been corrected (the processes involved had died but were still sticking around – ‘zombies’ – and had to be killed off completely so we could start things up again properly). That has been taken care of and things should be back to normal. We apologize again for the inconvenience this has caused and are looking into taking the necessary steps to prevent this from occurring again.


2:06 pm

Server Migrations (Round 3.5)

Posted (February 9th, 2010 at 2:06 pm PST) by Patrick

We’re continuing the migrations described here. We’re continuing with servers that we did not convert due to the eagles failure last week as well as a few others. The following servers will be rebooted some time after 6PM PST tonight:

falcons fay fever fire fitzpatrick flames floyd force forge gaborone galactus gambit genovese ghostrider giancana greengoblin greenlantern grizzlies haddock havana havok hawkeye hawkman hawks heat helsinki hera herediano highlanders hobgoblin hornets hulk humantorch huntress hurricanes indians islanders jaguars jazz jets johnnyblaze joker jubilee kabul kakuji kingpin kings kinshasa knicks lakers lapaz larchmont legs liberty libreville lightning lions lizards luanda luthor magic magneto malabo manama mariners marlins

The entire process will take a couple of hours and not all machines will go down exactly at six. Down time for individual machines will only be around 5 minutes. Check back here for updates, and as always, if you have any questions or concerns, please contact support!

– UPDATE Tue Feb 9 22:34:38 PST 2010 –

All machines are back up and running. If you’re having problems with your service on any of these recently converted machines, please contact support!


7:57 am

Iceman needs to chill out

Posted (February 9th, 2010 at 7:57 am PST) by Kyle

Iceman started exhibiting an ECC error on one drive while it’s RAID pair was in the process of rebuilding (the equivalent of your dog eating the backup copy of your homework while your making a new copy of the one he just ate). We will have to begin the process of preparing a new server to take it’s place and restoring it’s data from backups.

Update 11:00PST: Data is being served over the network while we are in the process of restoration, this takes time due the volume of data and having it take a day is not unheard of. Also, during the restoration, you may experience somewhat high loads on the machine. It’s part of the process, and we’re keeping an eye on it to make sure it doesn’t get out of hand. We appreciate your patience, and apologize for any inconvenience this may be causing.

Update 7:00PST 2/10/10: We are serving sites that have yet to be restored over the network and it looks like the backup server got bogged down.  This was causing an elevated load on iceman because disk operations were getting backlogged. The backup machine is back in good health and the load is dropping quickly on iceman.  Restoration is progressing but there are still many users left, we are unable to give a accurate timetable because the progress so far is skewed because of the previous slowness on the backup server – hopefully we will be able to soon.


10:05 pm

Nuggets got chomped

Posted (February 8th, 2010 at 10:05 pm PST) by Sandon

The web-server nuggets is currently down as it was throwing some file-system journaling errors and thus we had to restart it into single-user mode to run a fsck (file-system check). This process has been known to take anywhere from 1-5 hours depending on the amount of data/files on the server. More updates to come as they are available.

Update 3:20am Pacific: The file system check on this server has been completed and it is coming back online now!  We will be closely monitoring the server.  Please contact support if you are still experiencing any issues!

Update 10:45am Pacific: The machine has been rebooted and came back fine. If you’re still having issues, it’s most likely unrelated, so please send support a ticket with details. Sorry again about the inconvenience this downtime has caused!


6:41 am

pyongyang downtime

Posted (February 8th, 2010 at 6:41 am PST) by patrickm

We sincerely apologize for the downtime with the pyongyang server at this time. Our admin team is investigating the issue and hope to have services restored as quickly as possible.  Please watch this space for updates.

Update – 3:20pm PST: The server should be getting back to normal now. The filesystem on the server was having issues and became corrupted. We started our normal restoration plan, but after running several fscks it wasn’t repairing the data. We started to restore files from backup, but several other misconfiguration problems prevented that from happening. We’ve fixed all of those and data is now being copied over to the server from our backups. The server is now serving files from the backup server over the network and will switch back to serving the local files once the restore is complete. There should have been no data lost due to this issue and things should be running smoothly now. If you have any further issues with this server, please contact support.

Update – 6:30pm PST: It would appear that the file restores that are running are doing a fantastic job of driving the load up on the server.  This means some folks are still seeing downtime and 500 errors.  So “back to normal” isn’t all that normal at all.  Sorry about that!  More updates (hopefully) soon.

Update – 11:20am PST: Just wanted to touch base and let you know that the moves are still running. They are going slowly unfortunately but they are moving along. Sorry this has taken so long to resolve.

Update Feb 10th – 11:20am PST: We haven’t forgotten about this issue, we just don’t have any updates. The moves are still running and we won’t be able to do anything else until that completes. We will update this status post when the situation changes.

Update Feb 11th – 6:50 PST: We are roughly 2/3 of the way through restoration, hang in there!


11:13 pm

Network Maintenance – Sunday, Feb.7 11 pm – Monday, Feb. 8 2 am

Posted (February 7th, 2010 at 11:13 pm PST) by robertn

There will be a Network Maintenance Window tonight from 11 pm until tomorrow morning at 2 am.  We will be adding a Vlan for a new cluster in our downtown data center, increasing the  redundancy of our core architecture by restructuring our failover firewall, doing routine maintenance on our distribution switches and further solidifying our LAX datacenter’s new Vlan and cluster.

Thank you for your support.  If you experience any anomalous behavior, please contact Technical Support with details.

Maintenance completed


6:36 am

homie-vserver113 going down for RAID battery replacement

Posted (February 6th, 2010 at 6:36 am PST) by robert

homie-vserver113 will be going down for replacement of its RAID battery at about 7am PST today. Will be back up within 30 minutes. Affected PSes are:

ps17119 ps17161 ps17194 ps17252 ps17633 ps17697 ps17938 ps17944 ps18122 ps18130 ps18131 ps18139 ps18223 ps18224 ps18226 ps18230 ps18232 ps18233 ps18237 ps18238 ps18244 ps18245 ps18246 ps18247 ps18248 ps18249 ps18257 ps18258 ps18259 ps18273 ps18274 ps18275 ps18276 ps18277 ps18300 ps18301 ps18385 ps18386 ps18438 ps18578 ps18625 ps18626 ps18627 ps18628 ps18701 ps18708 ps18713 ps18722 ps18731

Update: Battery replaced and all PSes are up.


11:22 pm

Pinto is slowing to a crawl

Posted (February 5th, 2010 at 11:22 pm PST) by Sandon

The web-server ‘Pinto’ appears to have a problem with its hardware that has been causing various issues including the server crashing and rebooting. I went ahead and started the process to fail it over to a new set of hardware which typically takes 30 minutes to 1 hour to complete. More updates to come.

Update Feb. 6th 12:33 AM PST: Pinto is back up on its feet and running strong again and new shiny hardware. If you are having any issues with your sites and you are on ‘pinto’ please contact support.


 
© 1996-2010, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
12:25 pm

Server Migrations (Round 3)

Posted (February 4th, 2010 at 12:25 pm PST) by Patrick

We’re continuing the migrations described here. The following servers will be rebooted some time after 6PM PST tonight:

clippers cloak colossus colts constantine coonan copenhagen cowboys crew crusaders cummiskey cyclops d-fenders dagger dalitz deadpool deathlok dhaka diamondbacks dillinger djibouti dodgers dredd dynamo eagles earthquakes eastman eskimos exodus expos falcons fay fever fire fitzpatrick flames floyd force forge gaborone galactus galaxy gambit genovese ghostrider giancana greengoblin greenlantern grizzlies haddock havana havok hawkeye hawkman hawks heat helsinki herediano highlanders hobgoblin

The entire process will take 30-60 minutes, but down time for individual machines will be around 5 minutes. Check back here for updates, and as always, if you have any questions or concerns, please contact support!

– UPDATE Thu Feb 4 20:26:20 PST 2010 –

We’re about half way through the list of servers, but are halting the process until further notice.

Unfortunately, upon boot, one of the partitions on the server “eagles” came up with file system inconsistencies. We are checking the system now and will get it back up and running as soon as possible.

– Update Thu Feb 4 22:56:26 PST 2010 –

We’re back up and running on eagles. The filesystem errors have been corrected and all sites are up and running. If you have any questions or concerns, please contact support.


10:49 am

cannons going down for RAID replacement

Posted (February 3rd, 2010 at 10:49 am PST) by douglas

cannons will be going down for replacement of its RAID card at about 11:20am PST. Will be back up within 30 minutes, likely in half that time.

Update: RAID replacement occurred without incident. Except for the server room gremlin gnawing at my ankles. Nasty bugger.


6:33 am

Bugsy data mounting issue

Posted (February 3rd, 2010 at 6:33 am PST) by patrickm

We’ve discovered an issue with the way the Bugsy server is mounting user home directories. This is causing the websites hosted there to appear blank, or empty. Rest assured that no data loss has occurred. We are working on this issue and hope to have it resolved quickly. We thank you for your patience. Watch this post for updates.

Update: Issue resovled.


10:55 am

Server Migrations (Round 2)

Posted (February 2nd, 2010 at 10:55 am PST) by Patrick

We’re continuing the migrations described here. The following servers will be rebooted some time after 6PM PST tonight:

bills blackhawks blade blazers bluebombers bluejays bobcats bogota bonanno brasco brasilia braves brazzaville brewers bridgetown broncos browns bruins brumbies bruno brussels buccaneers bucks buenosaires bugsy bujumbura bulger bulls bullseye cable cairo canberra cannonball cannons capone cardinals castellano catwoman cavs cerebus

The entire process will take 30-60 minutes, but down time for individual machines will be around 5 minutes. Check back here for updates, and as always, if you have any questions or concerns, please contact support!

– UPDATE Tue Feb 2 20:05:01 PST 2010 –

All machines have been back up and functioning properly for a while now. If you have any problems with your service at all, please contact support.


7:03 pm

Server Migrations (Round 1)

Posted (February 1st, 2010 at 7:03 pm PST) by Patrick

We’re beginning the migrations described here now. The first set of servers to get the treatment are:

algiers alouettes angel angels apocalypse aquaman argonauts asmara avalanche avengers baku bandar banshee barker batman bayhawks bears beijing belmopan

The entire process will take 20-30 minutes, but down time for individual machines will be around 5 minutes. Check back here for updates, and as always, if you have any questions or concerns, please contact support!

– UPDATE Mon Feb 1 19:42:14 PST 2010 –

All machines are back up and functioning normally. If you are experiencing any issues with this subset, or any machine for that matter, please contact support.


6:11 pm

Server Architecture Migrations

Posted (February 1st, 2010 at 6:11 pm PST) by Patrick

Over the past two years, we have been using a network image to boot all newly deployed servers. This allowed us central management of our operating system image, as well as an easy upgrade path. Unfortunately, within the past few months, the software that we have been using to do this has become terribly unstable. As a result, over the next several weeks we will be converting machines back to a more traditional setup (where the operating system lives locally).

What this means for you:

Every server that is currently booted over the network will be rebooted at some point to come up on a local system. That’s it! It’s a fairly painless process that, to the end user, is nothing more than a typical reboot. Sites will experience roughly 5 minutes of downtime while the servers come back up. All conversions will occur after 6pm PST and be posted on dreamhoststatus.com.

Thank you for your patience and if you have any questions or concerns, please contact support.

The Happy DreamHost Netboot Slaughter Team


6:08 pm

Homie-vserver36 getting more RAM!

Posted (February 1st, 2010 at 6:08 pm PST) by Leon

Oh you lucky, lucky homie-vserver36 guests. You’re getting a pile of new memory to devour. Affected PSes are:

ps10813 ps10819 ps10828 ps11129 ps11244 ps11267 ps11625 ps11711 ps11832 ps11851 ps11862 ps11982 ps11992 ps12113 ps12141 ps12275 ps12511 ps12688 ps13008 ps13159 ps13180 ps18634 ps19815 ps20811 ps20880 ps20990 ps7584 ps7588 ps7596 ps7598 ps7601 ps7604 ps7612 ps7618 ps7630 ps7665 ps7680 ps7691 ps7703 ps7712 ps7782 ps7877 ps7880 ps7928 ps7934 ps7985 ps7998 ps8055 ps8124 ps8132 ps8180 ps8183 ps8186 ps8195 ps8233 ps8279 ps8770 ps9597 ps9606 ps9664

Should be back up momentarily!


4:53 am

homie-vserver140 raid degraded – FIXED

Posted (February 1st, 2010 at 4:53 am PST) by Ken

homie-vserver140 is having a raid issue and will be down while we work on it. We regret the inconvenience and will post updates as we find them out.

Affected personal servers are:
etch64-template.tgz  ps20436  ps20638  ps20682  ps20758  ps20829  ps20950
lost+found           ps20437  ps20639  ps20729  ps20763  ps20840
ps20386              ps20438  ps20640  ps20730  ps20793  ps20849
ps20395              ps20452  ps20641  ps20752  ps20814  ps20926
ps20396              ps20513  ps20672  ps20757  ps20828  ps20937

UPDATE 2010 Feb 01 0540am PST: Raid returned to service and P.S.’s returning.


11:48 am

Network Maintenance – January 31st, 2010 at 11 pm

Posted (January 31st, 2010 at 11:48 am PST) by robertn

There will be a network maintenance window between 11 pm Sunday and 2 am Monday.  Items to be addressed:

1.  Traffic shaping to alleviate congestion and improve performance
2.  Distribution switches trunk port maintenance – refining vlan traffic
3.  Edge to dist trunk updates to include Management vlans
4.  LAX  maintenance and introduction of  vlan for new cluster

This Maintenance was completed successfully.


10:20 am

Professorx having hardware issues

Posted (January 28th, 2010 at 10:20 am PST) by Justin@DH

Admins are looking into issues with the professorx server related to its raid. It is running a disk check and the admins are watching it right now, but unfortunately it will not be accessible until this check finishes. We will update this post when we have further info and when professorx is back up and running. Our apologies for the inconvenience.

Update: ProfessorX looks to be back in business! Please send a message to your ever-helpful DreamHost support cyborg if you are still having problems!

Update 2: I take it back, still having some issues. We’re working on it!

Update 3: Been up as of 2 hours ago. Sorry for the late update!


 
© 1996-2010, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
9:02 am

Server issues with homie-vserver106 and 141

Posted (January 26th, 2010 at 9:02 am PST) by Justin@DH

We are looking into issues with 2 PS host machines that are having issues booting. We are investigating the issues now to get them fixed and back up as soon as possible. This issue affects the following PS servers:

ps16217 ps16244 ps16245 ps16248 ps16249 ps16253 ps16262 ps16270 ps16271 ps16272 ps16289 ps16318 ps16320 ps16340 ps16341 ps16361 ps16377 ps16391 ps16403 ps16441 ps16478 ps16502 ps16528 ps16556 ps16601 ps16697 ps16718 ps16721 ps16723 ps16727 ps16781 ps16786 ps16843 ps16844 ps16849 ps16850 ps16854 ps16859 ps17091 ps17102 ps17127 ps17309 ps17373 ps17383 ps17410 ps17438 ps17535 ps17536 ps17577 ps17925 ps18217 ps18320 ps18321 ps18326 ps18327 ps18336 ps18349 ps18350 ps19769 ps19779 ps19790 ps19794 ps19972 ps20016 ps20119 ps20180 ps20212 ps20334 ps20622 ps20652 ps20685 ps20699
ps20361 ps20362 ps20375 ps20378 ps20384 ps20394 ps20398 ps20399 ps20420 ps20463 ps20514 ps20566 ps20593 ps20605 ps20610 ps20698

Update 5:08PM PST: We have began the restore process for homie-vserver106 Private Servers.  Depending on how much individual data is housed on each guest, it could take some time for your server to come back up. The following servers are still affected:

ps16217 ps16244 ps16245 ps16248 ps16249 ps16253 ps16262 ps16270 ps16271 ps16272 ps16289 ps16318 ps16320 ps16340 ps16341 ps16361 ps16377 ps16391 ps16403 ps16441 ps16478 ps16502 ps16528 ps16556 ps16601 ps16697 ps16718 ps16721 ps16723 ps16727 ps16781 ps16786 ps16843 ps16844 ps16849 ps16850 ps16854 ps16859 ps17091 ps17102 ps17127 ps17309 ps17373 ps17383 ps17410 ps17438 ps17535 ps17536 ps17577 ps17925 ps18217 ps18320 ps18321 ps18326 ps18327 ps18336 ps18349 ps18350 ps19769 ps19779 ps19790 ps19794 ps19972 ps20016 ps20119 ps20180 ps20212 ps20334 ps20622 ps20652 ps20685 ps20699

Update 9:55PM PST: As of this moment, the following Private Servers should be back up and running:

ps16244 ps16245 ps16248 ps16249 ps16253 ps16262 ps16270 ps16272 ps16289 ps16318 ps16320 ps16340 ps16341 ps16361 ps16377 ps16391 ps16403 ps16441

More should be coming back online shortly. We mostly wanted to make you aware of the progress before our graveyard shift took over. More updates soon!

Update 2010-01-27 3:33PM PST: The following Private Servers should be back up and running:

ps16244 ps16245 ps16248 ps16249 ps16253 ps16262 ps16270 ps16272 ps16289 ps16318 ps16320 ps16340 ps16341 ps16361 ps16377 ps16391 ps16403 ps16441 ps16478 ps16502 ps16528 ps16556 ps16601 ps16697 ps16718 ps16721 ps16723 ps16727 ps16781 ps16843 ps16844 ps16849 ps16850 ps16854 ps16859 ps17091 ps17102 ps17127 ps17309 ps17373 ps17383 ps17410 ps17438 ps17535

If your PS is listed and you’re having trouble still, please contact support so we can help you out. Once again, apologies for the trouble here. More PSs will be coming online as time progresses.

Update 2010-01-27 5:34PM PST: The following Private Servers should be back up and running:

ps16244 ps16245 ps16248 ps16249 ps16253 ps16262 ps16270 ps16272 ps16289 ps16318 ps16320 ps16340 ps16341 ps16361 ps16377 ps16391 ps16403 ps16441 ps16478 ps16502 ps16528 ps16556 ps16601 ps16697 ps16718 ps16721 ps16723 ps16727 ps16781 ps16843 ps16844 ps16849 ps16850 ps16854 ps16859 ps17091 ps17102 ps17127 ps17309 ps17373 ps17383 ps17410 ps17438 ps17535 ps17536 ps17577 ps17925 ps18217 ps18320 ps18321 ps18326 ps18327

Update 2010-01-27 8:39PM PST: The following Private Servers should be back up and running at this point:

ps16244 ps16245 ps16248 ps16249 ps16253 ps16262 ps16270 ps16272 ps16289 ps16318 ps16320 ps16340 ps16341 ps16361 ps16377 ps16391 ps16403 ps16441 ps16478 ps16502 ps16528 ps16556 ps16601 ps16697 ps16718 ps16721 ps16723 ps16727 ps16781 ps16843 ps16844 ps16849 ps16850 ps16854 ps16859 ps17091 ps17102 ps17127 ps17309 ps17373 ps17383 ps17410 ps17438 ps17535 ps17536 ps17577 ps17925 ps18217 ps18320 ps18321 ps18326 ps18327 ps18336 ps18349 ps18350 ps19769 ps19779 ps19790 ps19794 ps19972 ps20119 ps20622

All Private Servers on homie-vserver106 are now restored. If your PS is listed above and is not accessible, please write in to tech support. There were some lingering issues after the restoration process that will need to be cleared out in some cases. Thank you for bearing with us here as we work on getting things running properly again here.


4:36 pm

Luthor’s got more issues than Time magazine.

Posted (January 25th, 2010 at 4:36 pm PST) by Leon

But the main issue is that it’s RAID card is malfunctioning. We’re replacing this now. Should only be down for a few minutes while we swap hardware out.


11:25 am

Ongoing Hanjin (backup) server problems

Posted (January 25th, 2010 at 11:25 am PST) by danwy

As many of you have noticed, our backup server, hanjin, has not been very stable for the last several weeks. All attempts to try and resolve the issues have been slow due to the amount of data that is being stored on this server. The servers RAID configuration is acting up and thinks that one of the drives is double degraded. We are in the process of moving data to new hardware, but this issue is delaying the process. Because of this, there may be some intermittent issues until we’re able to get everything fully removed from the problem hardware.

The most recent issue manifested in users being logged into /root instead of their own directory and not able to access their files. These files are still on the drive, it just can’t be mounted by the servers filesystem. We have an admin in the datacenter today dealing with that specific issue in hopes of resolving it. Unfortunately, it’s probably going to be another month or so until the issues are fully resolved. We are doing all that we can to make this as smooth a process as possible, but all we can do is to wait for the huge amount data to move and try to keep things running in the meantime. We’re sorry for the inconvenience this has caused you. We’ll be sure to post an update when we have more info.


2:05 pm

Datacenter problems

Posted (January 24th, 2010 at 2:05 pm PST) by danwy

It looks like we lost connection to our LAX datacenter right now. We don’t have any details yet, but we are fully aware of the problem and admins have been scrambled to resolve the outage ASAP. We’ll update this post when we have more information.

Update 2:25PM: The connection to our LAX datacenter has been restored. We apologize for any inconvenience that you’ve experienced all services have restarted. If you notice further problems, please contact our support staff.


7:24 am

DH Panel Downtime

Posted (January 24th, 2010 at 7:24 am PST) by Justin@DH

We are currently investigating some issues with the DreamHost Control Panel not loading this morning. We are working to get this resolved and have an admin looking at it as I type this. Our apologies for the inconvenience caused while this is fixed.

Update – 8:30 am (PST) This should have been resolved for the last hour or so. This was due to an inode issue on our panel machines and has been fixed. Sorry for the the inconvenience.


4:15 pm

Network Maintenance Window Sunday January 24th, 2010 11 pm

Posted (January 23rd, 2010 at 4:15 pm PST) by robertn

This Sunday night from 11 pm until 2 am Monday morning there will be a non-intrusive, non-disruptive network maintenance window. We will be adding redundancy and cleaning up some legacy architecture from our completed move out Net2EZ.

Thank you for your continued support of Dreamhost.

This maintenance has been completed successfully.  We have added redundancy and capacity in the form of a second core.


10:20 pm

homie-vserver99 drive checks – DONE

Posted (January 19th, 2010 at 10:20 pm PST) by Ken

homie-vserver99 appears to be having some drive issues. We thank you for your patience while we look into this issue. Affected personal servers are:

ps15437 ps15720  ps15871  ps15951  ps16078  ps16132  ps16181  ps16398  ps16481  ps16820  ps17262  ps18197 ps15318 ps15461 ps15721  ps15878  ps15956  ps16082  ps16134  ps16185  ps16410  ps16482  ps16822  ps17315  ps19021 ps15350  ps15686  ps15753  ps15883  ps15962  ps16090  ps16140  ps16194  ps16419  ps16486  ps16839  ps17365 ps15356  ps15688 ps15777  ps15884  ps15969  ps16095  ps16144  ps16199  ps16420  ps16487  ps16861  ps17403 ps15361  ps15689  ps15778  ps15896  ps15978  ps16098  ps16150  ps16202  ps16429  ps16491  ps16867  ps17443 ps15366  ps15690 ps15780  ps15901  ps15983  ps16101  ps16151  ps16210  ps16442  ps16492  ps16901  ps17532 ps15371  ps15784  ps15906  ps15989  ps16103  ps16157  ps16212  ps16445  ps16496  ps16902  ps17602 ps15377  ps15691  ps15804  ps15910  ps15997  ps16106  ps16158  ps16304  ps16457  ps16593  ps16923  ps17628 ps15387  ps15697 ps15806  ps15918  ps16009  ps16108  ps16161  ps16362  ps16460  ps16641  ps16949  ps17701 ps15392  ps15699  ps15810  ps15923  ps16022  ps16112  ps16163  ps16384  ps16462  ps16739  ps16965  ps17746 ps15402  ps15702  ps15837  ps15928  ps16027  ps16113  ps16169  ps16388  ps16466  ps16762  ps17007  ps17793 ps15407  ps15854  ps15941  ps16032  ps16122  ps16172  ps16389  ps16470  ps16774  ps17081  ps17877 ps15411  ps15719 ps15864  ps15946  ps16033  ps16123  ps16178  ps16395  ps16471  ps16806  ps17176  ps17964

UPDATE Jan 19 23:08 PST: Checks are complete and the PS guests are coming back up. Thank you for your patience while we strive to improve your Dreamhost services.


10:00 pm

Panel Database Maintenance, January 19, 10PM PST

Posted (January 19th, 2010 at 10:00 pm PST) by DreamHost

We will be performing maintenance on our central database next week, beginning Tuesday, January 19 10PM PST, ending Wednesday, January 20 2AM PST. This will affect the following services:

DreamHost Webpanel
DreamHost-provided Webmail
Signups
Domain Registrations
DreamHost Wiki


12:59 am

Bugatti downtime

Posted (January 18th, 2010 at 12:59 am PST) by andrea

Bugatti is having hardware issues, so we’re moving the data to a new server. Sorry about the inconvenience this is causing. We’ll update/resolve this post as soon as possible.

UPDATE 4:15am PST: Bugatti has been successfully moved to a new server. However, at this point it’s being affected by the ongoing attack detailed at http://www.dreamhoststatus.com/2010/01/17/connection-issues-affecting-servers/

Very sorry about the continued inconvenience. Please check that post for updates.


6:36 pm

Connection issues affecting servers

Posted (January 17th, 2010 at 6:36 pm PST) by andrea

Our network is experiencing some kind of attack right now. Our admins are looking into this and working on resolution. We’ll post details and updates as soon as we get them, sorry about the inconvenience.

UPDATE 2010-01-18 12:15am PST: We’re still seeing some attacks, but they aren’t causing the sort of connection issues (slowness, downtime) as before. We’re keeping an eye on it, and will update/resolve this post when possible. Thanks for your patience.

UPDATE 2010-01-18 4:50am PST: Just a quick update that we’re still keeping an eye on this. Random issues may pop up here and there as it’s not completely resolved, but if in doubt don’t hesitate to contact our support team.

UPDATE 2010-01-18 1:15pm PST: The attack has mostly stopped, but we still have a few servers affected, so we’re not resolving this post until it’s all cleared up. As always, if you believe you’re affected or have any questions, don’t hesitate to contact our support team. Thanks for your patience.

UPDATE 2010-01-18 7:51pm PST: RESOLVED! Ok, we’re pretty sure this issues should be taken care of now. We’re setting this to resolved, so if you’re still having problems, you should not assume it’s related to this problem, and submit a support message asap! =)


 
© 1996-2010, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
4:47 pm

Network Maintenance Window Tonight

Posted (January 17th, 2010 at 4:47 pm PST) by Leon

Tonight (01-17-2010) beginning at 11PM PST, we will be performing a bit of network maintenance. This window will begin at 11PM PST tonight until 01-18-2010 at 2AM PST. We expect no interruption of service but we are letting you guys know as there is always a chance of a mishap! The changes will be to add new peering amongst a few other backend changes that, again, should have no downtime with any services. We apologize for the very short notice and hope you guys understand. If you have any questions, please contact our support staff.

This maintenance has been completed.  No downtime or interruption of service was caused by these procedures.

However, we are currently experiencing a DDoS attack that is causing disruption in certain services.  We are actively fighting this attack, and will have it mitigated shortly.  Thank you for your patience.


8:43 pm

Cloak file system errors

Posted (January 16th, 2010 at 8:43 pm PST) by DreamHost

Our shared hosting server “cloak” is currently down and undergoing a file system check. This can take a couple of hours to complete in some cases and all sites hosted on this server will be down for the duration of the check. We apologize for the downtime and will get it back up and running as soon as possible. More updates will be added as things develop.

UPDATE — Sun Jan 17 06:28:50 PST 2010 —
The file system check took forever but things should be back to normal now. Contact support if you are still encountering any issues.


6:28 pm

Upgrades this evening!

Posted (January 14th, 2010 at 6:28 pm PST) by Jason C

Customers on the looney, swarthy, postal, randy, spacey, frisky and blingy clusters may experience a minor amount of downtime this evening (see: just enough time for a reboot) due to software upgrades being performed by our admin team. These upgrades are being done to maintain stability and should begin around 12:00 AM, PST.

Updates should appear here as the night progresses.

Update 01/15/10 12:00am PST: Upgrade process has begun on the Randy cluster.

Update 01/15/10 12:45am PST: Upgrades complete on Randy cluster.

Update 01/15/10 1:31am PST: Upgrade process has begun on Postal cluster.

Update 01/15/10 2:00am PST: Upgrade complete on Postal cluster.

Update 01/15/10 2:40am PST: Upgrade process has begun on Blingy cluster.

Update 01/15/10 2:54am PST: Upgrade process complete on Blingy cluster.

Update 01/15/10 3:11am PST: Upgrade process has begun on Looney & Swarthy clusters.

Update 01/15/10 3:46am PST: Upgrade process complete on Looney & Swarthy clusters.

Update 01/15/10 3:51am PST: Upgrade process has begun on Frisky cluster.

Update 01/15/10 4:14am PST: Upgrade process complete on Frisky cluster

Update 01/15/10 6:00am PST: Upgrade process has begun on Spacey cluster.

Update 01/15/10 6:30am PST: Upgrade process complete on Spacey cluster.


5:24 pm

Dollar Got Mugged

Posted (January 14th, 2010 at 5:24 pm PST) by Leon

The dollar web server is currently experiencing issues with its hardware upon reboot. We’re working on getting it moved onto new hardware and it should be back up soon. Please check back for updates!

Update 8:28PM PST: Dollar is now back up on its new hardware and all services have been restored. If you’re still experiencing issues, please contact support!


4:42 pm

Homie-vserver106 hardware failure

Posted (January 14th, 2010 at 4:42 pm PST) by jeremy

A short time ago, one of our DreamHost PS hosts suffered a hardware failure that’s causing us to have to restore data from backups onto a new host. PS instances will come up one by one as their data is restored. A full list of affected guests follows:
ps16217
ps16244
ps16245
ps16248
ps16249
ps16253
ps16262
ps16270
ps16271
ps16272
ps16289
ps16318
ps16320
ps16340
ps16341
ps16361
ps16377
ps16391
ps16403
ps16441
ps16478
ps16502
ps16528
ps16556
ps16601
ps16697
ps16718
ps16721
ps16723
ps16727
ps16781
ps16786
ps16843
ps16844
ps16849
ps16850
ps16854
ps16859
ps17091
ps17102
ps17127
ps17309
ps17373
ps17383
ps17410
ps17438
ps17535
ps17536
ps17577
ps17925
ps18217
ps18320
ps18321
ps18326
ps18327
ps18336
ps18349
ps18350
ps19769
ps19779
ps19790
ps19794
ps19972
ps20016
ps20119
ps20180
ps20212
ps20334

UPDATE 2010-01-15 09:05 PM PST: The data restore has finished. Again, we sincerely apologize for any inconvenience this has caused and appreciate your continued patience. In the event that you are still having trouble, missing data, etc., please do not hesitate submitting a support request.


8:55 pm

Saprissa explains it all

Posted (January 13th, 2010 at 8:55 pm PST) by Jason C

As of about 12:10 PM [PST] today, our admin team has been working on fixing saprissa. Unfortunately, the downtime notice for today’s incident was appended to an old post. A new one should have been made — so we’re sorry for any confusion this may have caused.

Currently, the drives are doing a file system check. We have no time frame on when this will complete, but an update will be posted here when everything is back to normal.

Update 10:11 PM PST: Saprissa’s file system check has completed and all services on the server have been restored. If you’re experiencing any issues and you’re on the saprissa web server, please contact our friendly support staff!


5:00 am

ravens maintenance at 05:15PST Jan 13 2010 – DONE

Posted (January 13th, 2010 at 5:00 am PST) by Ken

Ravens will be brought down at around 05:15 PST to replace the raid card. Downtime should be less than 30 minutes.
We regret the inconvenience and appreciate your patience while we strive to improve your Dreamhost service.

UPDATE 05:15PST: Ravens is going down for maintenance. Should be back shortly.

UPDATE 05:35PST: Raid card replaced and all services restored. Thank you for your patience.


11:00 pm

Network Maintenance Window Sunday 2010-01-10

Posted (January 10th, 2010 at 11:00 pm PST) by Leon

On January 10, 2010, we will be performing a bit of maintenance on our network equipment located at the Downtown Los Angeles data center in order to “beef up” our internal network security. The window for this will begin on 01-10-2010 at 11PM PST and ending on 01-11-2010 at 1AM PST. This maintenance should result in zero downtime but as with any sort of changes being made there is always a slight chance of a mishap which is why we’re letting you all know.  If you have any questions, please feel free to contact our support staff.

– The Happy DreamHost Networking Team


8:32 am

Spawning a New Spawn

Posted (January 8th, 2010 at 8:32 am PST) by Kyle

Our shared hosting server Spawn isn’t working so well in the RAID (how your data is stored) department so we are migrating data to a new server which has a different controller, one that has been working out much better for our other servers which utilize it.  It is inconvenient but even more inconvenient would be having the current system eat it’s filesystem. I hope all you guys/gals can hold in there with us while we move to a better long term solution.  Luckily there isn’t a enormous amount of data that has to be moved and in the mean time we’ll be serving sites from network hosted backups (we would copy the data and then transfer services but have opted against that because we want to try to avoid writes to the current filesystem to make sure it’s integrity isn’t compromised). We will of course be providing status updates as we progress with the migration.


10:59 pm

ProfessorX file-system errors

Posted (January 5th, 2010 at 10:59 pm PST) by Sandon

One of our web servers (ProfessorX) is currently offline due getting file-system errors. It was rebooted and put in single user-mode so we could run a file-system check on it (currently in progress). This can take a couple hours to complete and all sites hosted on this web-server are off-line in the meantime. More updates to come.

Update 01/06/10 01:21 AM PST PorfessorX’s file-system check has completed and it is back up and serving files in a normal fashion. If you are still having issues with your sites and your web-server is listed as ‘professorx’ please contact support.


 
© 1996-2010, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
8:49 am

Saprissa needs to kick it

Posted (January 4th, 2010 at 8:49 am PST) by Kyle

The shared hosting server “saprissa” exhibited ext3 filesystem journaling issues this morning on the partition containing user data.  We have brought it down to single user mode while it undergoes a check to ensure data integrity.  This could be minutes or hours depending on how many inodes were part of the corrupted orphan linked list. We’ll provide updates once we have more information, sorry guys/gals!

UPDATE 2010-01-05 10:20AM PST: The machine was rebooted during the initial filesystem check. As a result the process did not complete. We are running the fschk again. It should be completed in a couple of hours ( worse-case scenario ). We sincerely apologize for the inconvenience this has caused. Your continued patience is greatly appreciated. Please check back for future updates.

UPDATE 2010-01-05 11:45AM PST: Just a quick update that the recovery is still going and will likely go into the afternoon. Very sorry about the inconvenience this is causing.

UPDATE 2010-01-13 12:10pm PST: Unfortunatly, we’re having issues with this machine again. It’s being worked on by our network admin. Please check back for updates, and sorry about the inconvenience again!


12:39 am

homie-vserver02 undergoing disk maintenance – Done

Posted (January 4th, 2010 at 12:39 am PST) by Ken

We appreciate your patience while homie-vserver02 is going through a drive check. Since this is an extended drive test, it will be down for a while and will update this status post as soon as we have more information.

UPDATE Jan 04 01:45PST: Drive test completed and homie-vserver02 is back up and running. Thank you for your patience.


2:35 pm

Mona is down

Posted (January 3rd, 2010 at 2:35 pm PST) by andrea

One of our web servers, mona, is currently down. Our admins are looking into it, and we’ll update this post as soon as there is progress in the case.
To check if you were affected, see your web server listed under Account Status in your panel.
Very sorry about the inconvenience this downtime is causing.

UPDATE 3:25pm PST: Mona is back online, thanks for your patience! If you’re having any more difficulties on that server, please contact support with the details.


2:58 pm

Homie-vserver106 undergoing maintenance

Posted (December 30th, 2009 at 2:58 pm PST) by Leon

D’oh! Looks like we need to run a filesystem check on this machine. This affects the following DreamhostPS:

ps16217 ps16244 ps16245 ps16248 ps16249 ps16253 ps16262 ps16270 ps16271 ps16272 ps16289 ps16299 ps16318 ps16320 ps16340 ps16341 ps16361 ps16377 ps16391 ps16403 ps16441 ps16478 ps16502 ps16528 ps16556 ps16601 ps16697 ps16718 ps16721 ps16723 ps16727 ps16781 ps16786 ps16794 ps16795 ps16796 ps16842 ps16843 ps16844 ps16849 ps16850 ps16854 ps16859 ps17091 ps17102 ps17127 ps17309 ps17358 ps17373 ps17383 ps17410 ps17438 ps17535 ps17536 ps17577 ps17925 ps18217 ps18280 ps18320 ps18321 ps18326 ps18327 ps18336 ps18349 ps18350 ps19769 ps19776 ps19779 ps19790 ps19794

Might be a couple hours before it’s done running. We apologize for the inconvenience!

Update: The file-system check completed about an hour ago and everything seems fine. All PS guests should be up and running but if you notice any issues, please contact support.

Update Jan 01 17:44 PST: Further file-system issues have been encountered. All PS guests are down again as the server is undergoing further maintenance. This may take a couple of hours. We apologize for the continued downtime here and are working to get this fixed ASAP.

Update Jan 01 22:10 PST: Homie-vserver106 is now back up and all guests services have returned. Again, we apologize for the downtime.


1:07 am

Scheduled Maintenance December 30 2009 at midnight – Done

Posted (December 30th, 2009 at 1:07 am PST) by Ken

We will be upgrading a power strip in one of our racks. The affected servers will be indians, reds, royals, marlins, twins, and yankees.

Scheduled maintenance time will be December 30 2009 around midnight PST and should last no more than an hour.

Thank you for your patience as we continue to strive to improve your DreamHost experience.

The DreamHost NOC guys.

UPDATE Dec 30 23:55PST: Reminder, will be bringing down the above servers to upgrade the power strip in about 15 minutes (Dec 31 00:15PST).

UPDATE Dec 31 00:20PST: Starting maintenance upgrade.

UPDATE Dec 31 01:50PST: All servers are back up and running on the upgraded power strip. Thank you for your patience as we continue to try and improve your Dreamhost experience. The NOC Guys


5:52 am

Scheduled maintenance tonight December 29 2009 at midnight

Posted (December 29th, 2009 at 5:52 am PST) by Ken

We will be upgrading a power strip in one of our racks. The affected servers will be grizzlies, pacers, rockets, spurs, and thunder.

Scheduled maintenance time will be December 29 2009 around midnight PST and should last no more than an hour.

Thank you for your patience as we continue to strive to improve your DreamHost experience.

The DreamHost NOC guys.

Update Dec 29 23:37PST: Just a reminder, we’ll be replacing a power strip in about 1/2 hour affecting the above servers.

Update Dec 30 00:05 PST: Bringning down servers to begin maintenance. Thank you for your patience while we work on improving your services.

Update Dec 29 00:40 PST: Power strip has been replaced an affected servers are back up and running. Thank you again for your patience while we continue to improve your Dreamhost services – The Dreamhost NOC Guys.


5:29 am

havana is having raid issues

Posted (December 26th, 2009 at 5:29 am PST) by Sandon

Havana is having an issue of losing its raid configuration and and having some file-system issues that requires someone to go to the data center and replace the controller and fix the problems. This is affecting you if your webserver is ‘havana’

Update 07:29 AM PST The raid controller has been replaced and an fsck is running. The file-system errors were on the OS volume not the customer data volume so it shouldn’t be that much longer to complete the fsck.

Update 08:29 AM PST Most of the websites on havana have been up for the last 20 minutes but a couple of the apache deamons had some issues starting up so a small percentage of sites were still down. The issues have been resolved and all services on havana should now be restored. Please contact support if you are on havana and still having issues with your site.

-Your Happy Late-night Holiday DH staff.


1:11 am

Peppy lost its pep.

Posted (December 22nd, 2009 at 1:11 am PST) by Sandon

The web-server peppy was throwing some FS errors so I had to temporarily take it down to run an FSCK (file-system check) on it. This process can take anywhere from 1-4 hours to complete. This is affecting you if your web-server is listed as ‘peppy.’

Update 5:52 AM PST: It looks like this is taking longer than expected. The fsck is on pass 1C which is the last pass and usually indicates its atleast 70% completed.

Update 6:26 AM PST: The FSCK is finished with its scan and it is writing the changes to the FS. This should only take 5 minutes or less and then the server will be rebooted and should be back up shortly.

Update 6:34 AM PST: Ok the machine has rebooted and all services have been restored. Please open a support ticket if you are on peppy and still having problems.


8:10 pm

Outgoing Mail From Web Servers Delayed

Posted (December 21st, 2009 at 8:10 pm PST) by Jason C

Users who utilize PHP’s mail() function (or sendmail) on their servers may not be seeing outgoing mail at the moment. It would appear that one of our filter machines is experiencing a bit of a mail clog. One of our admins has been tasked with clearing this up. As soon as there’s an update, we’ll post it here.

Update: This issue has now been resolved, as the mail queues on the affected server are now cleared up and processing normally! If you believe you’re still having an issue with this, please contact Support and they will be happy to investigate!

Update: At least one server’s mail queue is still having an issue.  We are addressing this and hope to have it resolved very soon!


3:01 pm

Dead Powerstrip = Dead Machines

Posted (December 21st, 2009 at 3:01 pm PST) by Leon

Well, temporarily anyway. The affected machines are:

chip wario andross dedede psmysql16760 psmysql16819 psmysql3623 psmysql4248 psmysql4285 psmysql4294 psmysql4307 psmysql5720 psmysql5954 psmysql6085 psmysql6312 psmysql6314 psmysql6603 psmysql6948 psmysql6994 psmysql7170

We’re replacing the power strip right now and the machines should be back shortly. Apologies for the inconvenience!

Update: Everything’s back up and the strip has been replaced. Thanks for your patience! Please send a message to your local dreamhost techbot for great justice!


 
© 1996-2009, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
12:00 pm

Ruby gem updates

Posted (December 21st, 2009 at 12:00 pm PST) by AndrewF

We will be upgrading the following Rails-related Ruby gems on Monday:

  • rails / actionmailer / actionpack / activerecord / activeresource / activesupport (all to 2.3.5)
  • rack (to 1.0.1)

As well as: Previous updates available after the jump →


8:16 am

The Unwritten Journals of Peppy

Posted (December 20th, 2009 at 8:16 am PST) by Kyle

Peppy aborted journaling on the filesystem containing user data and remounted in a read only state.  To preserve data integrity on this machine we have brought down to single user mode whilst it undergoes filesystem check to rectify the problem.

Update 8:23 PST: The machine is back up but we suspect the underlying RAID card may need to be replaced.  Services are currently restored, prior to the replacement we’ll post a update here.

Update 13:25 PST 12/21/09 We will be changing the machines RAID card out soon, this is only for the machine Peppy. We will post here once the machine has been restored.

Update 14:13 PST 12/21/09 Peppy is now backup, but we are forgoing to RAID card swap while we investigate the server further. If you notice any problems please contact our support staff through your panel.


6:56 am

Hanjin sprung a leak

Posted (December 20th, 2009 at 6:56 am PST) by Kyle

One of the machines which contains data for Hanjin (customer uploaded backup machine) took a dirt nap this morning.   We are working on getting it back online but some customers will not be able to access backup data, we are truly sorry for the inconvenience.

Update 11:05AM: This should be taken care of now, we’ve had our system administrators review one of the volumes on Hanjin that was causing the issue. If you’re still experiencing problems, please contact our support staff.


2:00 pm

hanjin ram upgrade! 2009-12-18 2pm pacific

Posted (December 18th, 2009 at 2:00 pm PST) by kitchen

To alleviate some of the issues we’ve been having with the hanjin server we will be taking it offline for approximately 15 minutes at 2pm pacific time on 2009-12-18 to put a lot more ram in it, hopefully allowing the machine to keep a bigger metadata cache and thus allowing for higher throughput for any process which does any directory listings (which is pretty much anything any of you folks use on the server)

We will post an update here once we start, and another one once we’re finished.

Update @ Fri Dec 18 14:01:28 PST 2009: we are beginning now. We’ll update once we’re done!
Update @ Fri Dec 18 14:34:02 PST 2009: we’re all done! The machine has actually been up for about 20 minutes, but we didn’t want to post until we were fully satisfied everything was back to normal.


8:20 am

MySQL phillip server rebooted

Posted (December 18th, 2009 at 8:20 am PST) by patrickm

This morning, the phillip MySQL server’s RAID controller was rejecting I/O.  This unfortunately required that we reboot the server in order to resolve the problem.  We try to avoid this at all costs, due to the likelihood that certain tables in MySQL databases will become corrupted.

After the reboot, some of the services took a little longer than expected to come back up, but the server is humming along once again!

We immediately grabbed copies of all the database backups for this server, so that we can restore databases for any customers that are experiencing corruption after this reboot.  That way, the normal rotation of MySQL backups won’t overwrite the good ones with corrupt ones!

If this is your MySQL server (which you can see by clicking the “Account Status” drop-down in your Web Panel) and you are still experiencing database issues this morning, please contact support so that we can investigate and get that restored and fixed for you!

Also note that we have to do the restores ourselves and the “Restore Database” button on the panel will not do the trick in these instances.


5:10 am

Bengals lost its stripes.

Posted (December 17th, 2009 at 5:10 am PST) by patrickm

We’re sorry to report that the shared-hosting server Bengals is down at this time.  We’re working to get it back up as quickly as possible.  It appears to be RAID issues and we are swapping the controller. Please check back here for updates!

Update 6:48 PST: We coaxed the raid back up and it’s rebuilding 1 drive, disk IO will be impacted during the rebuild but it shouldn’t be very noticeable.

Update 9:28 PST: Sadly, the RAID freaked out again and we are in the process of trying to recover it. In the meantime the server will be completely inaccessible.

Update 9:48 PST: It’s grim, it seems that simultaneous drive death on a RAID pair occurred, defeating our RAID 1+0 redundancy.  We are prepping a replacement machine and will be restoring customer data from backups in the most expeditious manner as possible.

Update 11:30 PST: Had some hiccups provisioning the new hardware but we are past that hurdle now and have begun our restoration process which will serve customer data over the wire directly from the backup server while data is being restored to the local disk.

Update Dec. 18 8:15 PST: Roughly 10% of users have been fully restored, this number would have likely been more had the restore not halted early this morning because the rsync service gave up on us on the backup machine.  Hold in there it’s chugging along.


2:43 pm

Warmachine Tanked

Posted (December 16th, 2009 at 2:43 pm PST) by Leon

Looks like warmachine is having some hardware issues. We have technicians working on fixing it now. We will update as soon as it’s back online!

Update: Looks like warmachine is back. Please report any outstanding issues to your nearest DH technically-oriented support person.

Update: 8:51pm PST The machine is up and running, however some users may notice data loss (or permission denied errors). We’re restoring those users, and are about 30% done. We should be done within the next hour or two.


11:47 am

Private Server downtime.

Posted (December 15th, 2009 at 11:47 am PST) by mir

A couple of our Private Server hosts just froze up on us. We have rebooted them and they are already coming back up. We are investigating a stack trace from it that we caught in one of the log buffers in an effort to prevent any similar occurrences in the future.


4:44 pm

Hug needs a…yeah, I’m going for it: hug.

Posted (December 14th, 2009 at 4:44 pm PST) by Leon

Looks like the power supply burnt out. We have technicians replacing the failed hardware right now. Thank you for your patience!

Update: It’s back up. We tried another old machine and that didn’t work either. Ended up just taking the whole thing apart and swapping some goofiness out. Please send a message to your friendly DH support robot if you require any additional assistance!


3:55 pm

Quick Network Maintenance

Posted (December 12th, 2009 at 3:55 pm PST) by Leon

We will be performing quick network maintenace tomorrow night, December 13, 2009, beginning at 11:30pm PST with the maintenance window lasting until 1am PST. This maintenance shouldn’t cause any downtime as we are just adding some configurations to our network equipment to better assist us with monitoring traffic.  We will be onsite for this change just in case anything does go haywire but again, we do not expect any downtime with adding these monitoring configurations. If you have any questions, please contact our support staff.

– The Happy DreamHost Networking Team

UPDATE Dec 14th 12:39am PST: Maintenance has been completed with no apparent downtime. Thank you for your patience while we continue to strive to give you the best Dreamhost service.

– The Happy DreamHost Networking Team


 
© 1996-2009, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
10:42 am

Network Maintenance Window

Posted (December 9th, 2009 at 10:42 am PST) by micah

Our datacenter staff have identified a loose connection on a specific card that handles one (out of three) of our upstream providers. They have RMAd the DWDM and will be replacing it tomorrow after midnight.That’s Friday, December 11, beginning at 12:01am and lasting approximately 1 hour.

Because it is only one provider, out of the three, there should be no impact from this downtime. We have successfully tested this scenario in the past without so much as a blip in performance for our customers. The other 2 providers easily pick up the balance of the bandwidth.

Again, we do apologize for our network issues of late, we find them unacceptable as well, and while a maintenance window is probably the last thing everyone wants to hear about from us this week, it really is a preventative measure in our effort to avoid any more customer facing incidents.

Happy DreamHost Loose DWDM Team!


5:05 am

Another DDOS run happening

Posted (December 9th, 2009 at 5:05 am PST) by Ken

At about 04:30am, another attack was detected on one of our servers. We’re actively working on a resolution and hope to have full services running again.

Following servers appear to be affected:

flames, thrashers, lightning, predators, blackhawks, islanders, monarchs, liberty, fever, shock, earthquakes, dynamo, rapids, crew, fire, saprissa

Again, we’re sorry for the inconvenience and hope to have a resolution shortly.

UPDATE 05:28 AM PST: Disabled the IP that was receiving the attack. Services should be returning to normal now for these servers.


11:23 pm

Network Outage

Posted (December 8th, 2009 at 11:23 pm PST) by Sandon

It looks like our core router is having trouble again. Our cisco/network guy is on his way into the data center to troubleshoot the issue.

It looks like this is affecting all machines on our network. More updates to come.

Update 11:51 PM PST: The network is back up. More details to follow.

Update 12:02 AM PST: While working on a DDoS in progress, the core router Access Control List was being worked on to mitigate the problem. During the updates, a temporary outage was experienced, but is now back to normal. We regret the inconvenience and appreciate your patience while we strive to improve your Dreamhost experience.


5:56 pm

Brandine being moved to new hardware.

Posted (December 7th, 2009 at 5:56 pm PST) by Leon

Man, it’s hard to find quotes for her that work in this context. Anyway, the hardware died and a new machine is being set up. Should be about an hour before it’s all back up and running. Sorry for the inconvenience!

Update 19:00 PST: The move has finished and all services are reporting back up. If you still notice further issues, please contact our support staff.


1:20 pm

Network Outage

Posted (December 7th, 2009 at 1:20 pm PST) by Dallas

DreamHost is currently experiencing a networking outage, affecting all services. We will update this page with details as we get them.

Update 1:45 PM: The network is up and working again at the moment. We are investigating to find the root cause. Preliminary information suggests an OSPF failure.

Update 2:04 PM: We have had several network outages recently and we know that’s not acceptable. We are in the process of reviewing our current network management procedures and planning for the changes we’ll need to make to allow for the future.


7:53 am

Westside has Beef!

Posted (December 4th, 2009 at 7:53 am PST) by Leon

One of our filers, westside, has been experiencing issues this morning. Users mounted on this filer will be unable to reach their data until we are able to bring the filer back up. The following machines are affected and users on them can/will experience high load or unavailable data while we work on the issue:

bob bomberman dollar droop fidget gage ikari quickbrownfox rosco scipio

Update: westside was not our newest hardware so we have been offloading it – users are up and running in the meantime so everything should be working (though there might be slightly higher than normal loads caused while we finish that up – it should be completed at the end of this weekend).


7:48 am

Mail Woes!

Posted (December 3rd, 2009 at 7:48 am PST) by mir

Customers have been noticing issues with mail sending and webmail responsiveness.

Mail sending:

We’re still working on networking issues that have cropped up after our data center consolidation move and some saturation this morning has caused issues with our mail machines in the homie cluster.  We are looking into adding capacity to help alleviate these woes.

Webmail:

Several of our webmail machines are being nitpicky this morning and it is being investigated by our admin team. This should hopefully be resolved shortly.

Update 9:37 PST: Our webmail machines mount a filer that recently had several disks explode, the disks have been replaced but things will be slow (mainly attachments) while it’s rebuilding those volumes.  We are at 7% right now and will continue to provide updates as it progresses.  This filer is also connected to wiki.dreamhost.com and our dreambook service so things will be sluggish there as well. Sorry folks!

Update 10:02 PST: Webmail is back up and running fine now that we have disabled attachments, the wiki will be slow on loading images but should otherwise be speedy. Rebuild is at 11%.

Update 13:05 pm PST: One volume is beating the other out at 37% while the other is still at 26%

Update 14:02 pm PST: To clarify an earlier statement, attachments in webmail are going to be disabled while the volumes rebuild on the troubled filer.  This means that if you’re expecting an important attachment and rely on our webmail services, it might be a good idea to set up a desktop based IMAP client.  Once the volumes have been rebuilt and things are stable — which shouldn’t happen any later than this evening — we will post a final update here.

Update Dec.4 9:57 PST: Both volumes have reconstructed and attachments have been reactivated for webmail.


11:30 pm

Homie-vserver106 raid controller died

Posted (November 30th, 2009 at 11:30 pm PST) by Sandon

One of the vserver hosts is no longer detecting its raid controller preventing the machine from booting. This is affecting you if you have a PS named one of the following:

ps16217 ps16244 ps16245 ps16248 ps16249 ps16253 ps16262 ps16270 ps16271 ps16272 ps16289 ps16299 ps16318 ps16320 ps16336 ps16340 ps16341 ps16361 ps16377 ps16391 ps16403 ps16441 ps16478 ps16502 ps16528 ps16556 ps16601 ps16697 ps16718 ps16721 ps16723 ps16727 ps16781 ps16786 ps16794 ps16795 ps16796 ps16842 ps16843 ps16844 ps16845 ps16849 ps16850 ps16851 ps16852 ps16854 ps16859 ps17091 ps17102 ps17127 ps17309 ps17358 ps17373 ps17383 ps17410 ps17438 ps17535 ps17536 ps17577 ps17925 ps18217 ps18280 ps18320 ps18321 ps18326 ps18327 ps18336 ps18349 ps18350

One of our on-site staff is working on the issue right now.

Update 11:56 PM PST The raid controller has been replaced and all the guests are back up and running. Please contact support if you are on one of the above PS’s and still having issues.

Update 2:04 AM PST Unfortunately, this host server is now undergoing a file-system check, which is keeping the server (and all guests above) down.  This shouldn’t take too long and we’ll update this post when it has completed!

Update 2:38 AM PST We’re happy to report that the file-system check on this host server has completed!  All the guest servers (listed above) are up and running again.  We’ll continue to closely monitor for any new issues, but please contact Support if you still have any issues!

Update 3:11 AM PST Since the last update, the RAID card on the server failed.  We have now replaced this card once again and are waiting on a file-system check to complete now.  We will post when this is done and will also be investigating this incident thoroughly, as 2 card replacements in one day shocks us, too!

Update 3:25 AM PST File-system check has completed and all guests are back up and running.  As usual, we’re watching this one closely and will update if we have any other issues!


7:42 pm

Network Failure

Posted (November 30th, 2009 at 7:42 pm PST) by Sandon

It looks like we are having a problem again with our network which appears to be reminiscent of the failure we had last week. More details to come.

Update 08:05 PM PST: It looks the network problems which caused packetloss and some other problems for around 20 minutes or so were caused by some problem from moving links around on some distribution switches which went from strait through -> cross-over which caused some vlan interfaces to flap and the routes had to be shut down at our core and the affected cables fixed. Everything should be back to normal now.

Update 09:35 PM PST: It looks like everything is back up network-wise but there were some residual effects of the network problems causing machines to lose their connections to their file-servers in our LAX datacenter. Some servers are still experiencing downtime which are being rebooted and investigated in the following clusters:

frisky, blingy, spacey, swarthy

Update 10:40 PM PST: Just about all the residual affects of the network outage have been recovered and service should be back up to normal.


1:14 pm

Network Flap

Posted (November 30th, 2009 at 1:14 pm PST) by Ralph

We are currently experiencing a network problem at one of our data centers. We are actively investigating. More information will be added as we get it.

Update: This issue is now resolved. There may be some minor issues sticking around. If so just let us know via support and we’ll help you. There was a configuration issue when our data center switched over their network to our core router. This caused one of the data centers to go down and was immediately fixed by our data center staff. The network then returned to normal roughly 1:30pm PST, with a few issues to clean up after the outage.


 
© 1996-2009, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
11:31 am

Intermittent webmail errors when sending.

Posted (November 26th, 2009 at 11:31 am PST) by mir

We are terribly sorry for the inconvenience during the holiday but a pair of our webmail servers are having storage space issues. We’re working on clearing out space but in the meantime customers are intermittently seeing the following error when trying to send messages:

‘452 insufficient system storage’

This should be cleared up soon (we’ll update this post when it is fixed for good) but in the meantime you can avoid it by using a normal non-web-based client as our regular mail servers are not involved in the issue and are operating without problems.

Update: 2009-11-26 13:20 PST:

This issue has been resolved – we’ll keep an eye on it to be sure but we have plenty of space available now on the effected servers.


6:33 pm

Reminder: Network Hardware Maintenance and Moves!

Posted (November 25th, 2009 at 6:33 pm PST) by Ralph

Just a reminder that tonight we have hardware moves going on at 8pm. The information is posted in the following status post:

http://www.dreamhoststatus.com/2009/11/24/network-hardware-maintenance-and-moves/

Also beware of server reboots also expected to start around 8pm:

http://www.dreamhoststatus.com/2009/11/25/server-reboots/

These reboots are exciting for us as we will be implementing ksplice. This service allows us to make back end server changes and updates without rebooting our machines, which means less downtime. If you are interested you can read more about ksplice here: http://www.ksplice.com/

You may also want to catch up on our blog:

http://blog.dreamhost.com/2009/11/24/thanksgiving-travel/

This details our recent data center move which has caused some amount of support as we moved and adjusted our network to another data center. This will give you an idea of the hard work our administration team put in for you.

Happy thanksgiving everyone and we’ll do our best to keep your sites up.

Update: 9:20pm PST The switch over has started on time and the NAT change has taken place. We are working on any machines that were having trouble after the move. The reboots are next.

Update: 10:20pm PST First 50 servers in our reboot are underway:

addisababa algiers alouettes angels aquaman argonauts asmara avengers baku bandar banshee barker bayhawks bears beijing belmopan bengals bills blackhawks blazers bluebombers bluejays bobcats bogota bonanno brasco brasilia braves brazzaville brewers bridgetown broncos browns bruins brumbies bruno brussels buccaneers bucks buenosaires bugsy bujumbura bulls

Update: 10:35pm PST Second set of servers are now going. First group should be up now.

cairo canberra cannons capone cardinals catwoman cavs chivas clippers cloak colossus colts constantine coolblue coonan copenhagen cowboys crew crusaders cummiskey cyclops d-fenders dalitz deadpool dhaka diamondbacks dili dillinger djibouti dodgers dynamo eagles earthquakes eastman eskimos expos falcons fay fever fire fitzpatrick flames floyd force gaborone galactus galaxy genovese greengoblin greenlantern grizzlies havana hawks heat helsinki herediano highlander hornets hulk hurricanes iceman indians islanders jaguars jazz jets jubilee kabul kings kinshasa

Update: 10:58pm PST Last set of reboots on the way: (any machines that did not come right up are getting immediate attention.)

havana hawks heat helsinki herediano highlanders hornets hulk hurricanes iceman indians islanders jaguars jazz jets jubilee kabul kings kinshasa knicks lakers lapaz legs liberty libreville lightning lions lizards luanda luthor magic magneto malabo manama mariners marlins mclean mets minsk monarchs moroni nassau nationals ndjamena netboot-testy nets nicosia nightwing niners nuggets obelix oilers olympiacos ottawa ouagadougou outlaws pacers packers padres panthers paris patriots penguin penguins phillies phnompenh pirates pistone pistons porto-novo prague praia preacher predators pride quito raiders rams rangers rapids raptors ravens rays reds redwings revolution robin rockets rockies roseau roughriders royals sanjose sansalvador santiago santodomingo saprissa sarajevo seahawks shadow shock snowbird sofia sox sparks spiderman spillane spurs stampeders steelers stjohns suns suva tallinn texans thimphu thrashers thunder tigercats tigers tirane titans trafficante twins venom vienna vikings walsh waratahs warriors wizards yamoussoukro yankees yaounde yerevan zagreb zwerbach

Update: 11:15pm PST Looks like we missed a few machines in our post. Sorry about that. Here they are:

angel apocalypse avalanche batman bishop blade bucks bujumbura bulger bullseye cable cannonball castellano cerebus dagger dredd exodus forge galactus gambit ghostrider giancana grizzlies haddock havok hawkeye hawkman hobgoblin humantorch huntress johnnyblaze joker kakuji kingpin marv masseria mekon mothman mrfantastic mystique namor nitti phantom professorx proty quicksilver riddler rogue sabretooth sandman silversurfer spawn spectre starfire storm sugeknight sunspot supergirl swampthing terranova thing thor tick twoface warmachine wasp wolfsbane wonderwoman zatara

These are currently getting rebooted.

Update: 11:50pm PST Okay all the reboots have finished finally. If you are noticing problems let us know via support. Sorry for the problems.


2:03 pm

Shared Hosting and Private Server reboots happening today

Posted (November 25th, 2009 at 2:03 pm PST) by patrickm

We unfortunately need to quickly reboot all Shared Hosting servers and Private Servers today. We apologize for any inconvenience that this may cause. This should cause minimal downtime for web hosting services.

Please check back here, as we will post more information about this maintenance once it is completed.

Update: The reboots are currently set to start at around 8:00pm PST. As we have more information we will update this status post.


1:31 pm

Postal file server move! 2009-11-28 8pm pacific

Posted (November 25th, 2009 at 1:31 pm PST) by kitchen

As part of our rack consolidation efforts, we need to move one of postal’s file servers: wok. This will cause anyone who has mail or data on this machine to have some brief downtime. We hope to have this done quickly, but we’re going to schedule 1 hour of downtime, just in case.

Users and services on the following servers will be affected:

absinthe achilles admiralty aero annie apu araisoft bandini barranca bart baseline bellflower beverly bigredgiant bitters blowpop bollocks bonbon bounty boyd brandine brawndo broztrain bubbleyum burns butler cahuenga caitlin caramello carl catalina cerritos chalmers cheeky chex cletus cookie crenshaw damage dawber dib drumstick fireballs flanders fundip garland gaz gimli gir glass gloop gobstopper goober gotcha granola harpo hoho homer jurupa kearny krusty larchmont larry leancode lenny lifesaver lisa lovejoy lucas madsamurai marge mcclure mikaeljansson mildred millhouse monkey monroe moorpark mounds necco nelson nou nougat ocean ohso olive olympic otto outside ouzo patagonline pico piggy pogo powerbar puch raiden ralphie redondo rizzo roscoe salt saltine sanjay sawtelle scooter scotch seamus selma shemp sidestep skinner smithers spree sunset sweetdreams taffy tak talkbox tinyblip tnmedia toblerone torrance truffle twizzler vanowen washington whittier wiggum willie wilmington wilshire yerba york zim zuma

We’ll post updates here when we start, and when everything is finished!

Update: 8:14pm pacific: we are starting this now. We expect around 15 minutes of downtime, hopefully less!

Update: 8:37pm pacific: we have finished the move and services should all be back up and running. Total downtime for the file server was 642 seconds. If you’re having any issues, please contact support!


11:31 pm

Network Hardware Maintenance and Moves!

Posted (November 24th, 2009 at 11:31 pm PST) by Leon

As part of our recent datacenter/server moves, we will be moving our networking equipment tomorrow (11-25-09) at 8pm Pacific time. Specifically, we are moving our Cisco ASA’s which provide our Network Address Translation (NAT). This means you will experience brief downtime with your websites/MySQL/email/webpanel access while they are moved and reconfigured. We expect this downtime to only last about 15 minutes; however, we are making the downtime window approximately 3 hours in order to leave room for any collateral damage that may occur. If you have any questions, don’t hesitate to contact our friendly support staff!

Edit 11-25-09 10:40am Pacific: Whoops! I made a typo! It’s today 11-25-09 at 8pm Pacific that these moves are going to happen. Sorry!

– The Happy DreamHost Networking Team


12:31 am

Network Failure

Posted (November 24th, 2009 at 12:31 am PST) by micah

DreamHost is currently experiencing a fairly large network failure. The extent is unknown at this time however it seems that most of our central services (dreamhost.com, panel.dreamhost.com, webmail, etc…) as well as our customer sites and email are affected.

We have our network experts up and looking into the situation right now, hopefully a solution or at least more information will be forthcoming.

Update 3:20am Pacific - The issue with the network has now been resolved!  At this time, our network and most services are back up and running.  We’re still running triage on the servers and working to bring everything back up to full functionality.  If you’re still experiencing any downtime, know that we’re working on those issues.  Feel free to contact us for more info and please check back with this post for more updates when we have them.

Update 11:45am Pacific - Unfortunately, we are still having some lingering networking issues that are manifesting in various ways: domain and mysql downtime, email connection and sending issues. Our admins are looking into this. We apologize for the inconvenience!

Update 11:45am Pacific - We believe the earlier problems are now gone. If you’re still experiencing difficulties with your hosting service, please submit a support ticket with details, so we can look into it for you.


2:09 pm

Homie-vserver99 is moving to new hardware

Posted (November 23rd, 2009 at 2:09 pm PST) by andrea

Our homie-vserver99, host of many VPS accounts has been having issues since yesterdays, so we decided to migrate it over to new hardware.

The affected list of PS is as follows:

ps15318 ps15329 ps15350 ps15356 ps15361 ps15366 ps15371 ps15377 ps15382 ps15387 ps15392 ps15402 ps15407 ps15411 ps15437 ps15461 ps15555 ps15686 ps15688 ps15689 ps15690 ps15697 ps15699 ps15702 ps15712 ps15720 ps15721 ps15753 ps15777 ps15778 ps15780 ps15784 ps15804 ps15806 ps15810 ps15837 ps15854 ps15864 ps15871 ps15878 ps15884 ps15896 ps15901 ps15906 ps15910 ps15918 ps15923 ps15928 ps15941 ps15946 ps15951 ps15956 ps15962 ps15969 ps15978 ps15983 ps15989 ps15997 ps16009 ps16022 ps16027 ps16032 ps16033 ps16078 ps16082 ps16090 ps16095 ps16098 ps16101 ps16103 ps16106 ps16108 ps16112 ps16113 ps16122 ps16123 ps16132 ps16134 ps16140 ps16144 ps16150 ps16151 ps16157 ps16158 ps16161 ps16163 ps16169 ps16172 ps16178 ps16181 ps16185 ps16194 ps16199 ps16202 ps16210 ps16212 ps16304 ps16362 ps16384 ps16388 ps16389 ps16395 ps16398 ps16410 ps16419 ps16429 ps16442 ps16445 ps16457 ps16460 ps16462 ps16466 ps16470 ps16471 ps16481 ps16482 ps16486 ps16487 ps16491 ps16492 ps16593 ps16641 ps16681 ps16712 ps16739 ps16762 ps16774 ps16806 ps16820 ps16822 ps16839 ps16861 ps16867 ps16902 ps16923 ps16949 ps16965 ps17007 ps17081 ps17262 ps17315 ps17365 ps17403 ps17443 ps17532 ps17602 ps17628 ps17746 ps17793 ps17877 ps17964 ps18197

We estimate the downtime for the migration to be around an hour, during which you won’t be able to access your server or load your sites. We apologize for the inconvenience and we’ll update the post as soon as progress is made.

UPDATE 4:45pm PST: The migration went well, just took a bit longer than expected. We restarted all the virtual private servers on that machine, and are currently reconfiguring the network as last step. We’ll update again when it’s complete, just wanted to get a heads up due to the longer wait. Thanks for your patience!

UPDATE 4:55pm PST: All the private servers listed above should be up at this point. Let support know if you’re still having issues, with details, and we’ll take a look for you.


5:01 am

Riddler down

Posted (November 23rd, 2009 at 5:01 am PST) by patrickm

Due to configuration issues with the server, we are unable to complete a reboot at this time.  We’re investigating this and hope to have the problems resolved shortly.  Unfortunately, this issue is causing the server to remain down at this time.  We will be updating this post as soon as we have more information on this problem.  Thanks for your patience and we sincerely apologize for the inconvenience.

UPDATE: riddler is now up and services restored. Sorry for the inconvenience.


3:42 pm

Brandine getting new hardware.

Posted (November 21st, 2009 at 3:42 pm PST) by mir

We apologize to the users on the server Brandine – the hardware has failed so we’re having to move everyone onto a new hosting machine (nothing else will change though – just new physical hardware so the name and data and everything else will remain the same). We should have this completed in the next hour or so and will update this post once it’s completed.


11:18 am

A handful of customers missing mail data temporarily.

Posted (November 21st, 2009 at 11:18 am PST) by mir

A small number of accounts had mail that was unable to be moved over to new hardware before we had to power down the last of our servers and move them out of the old data center. For now we are setting up those users (without the data) on the new system so they should still be functional and we’ll be working on restoring the data once we have moved the servers (the data is not lost, just not currently available). We apologize for this – we understand how important mail is and wanted to reassure those effected that we’re aware of the matter and are working to correct it.

UPDATE 2009-11-24 11am PST: The restores have commenced last night. We won’t resolve this post until they are all done, this is just a heads up that we’re making progress in this case. We appreciate your patience!

UPDATE 2009-11-28 14:00 PST: We have restored data from approximately 1/3 of the file servers that had to be taken down. We should be up to 1/2 by the end of the weekend and will be plugging in the rest of the filers to get their data after that. We apologize that this is taking longer than originally estimated but are getting this executed as quickly as we possible can with the resources we have available.


 
© 1996-2009, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
9:54 am

Wolverine declawed

Posted (November 19th, 2009 at 9:54 am PST) by patrickm

We’re sad to report that the wolverine server is experiencing some issues returning from a reboot we needed to complete this morning.  We’re working on this right now and will update this post as soon as we have additional information.

UPDATE: We truly apologize for the lack of updates on this post.  Due to the central services migration from last night, this post slipped past us.  The great news is that this issue is completely resolved.  We found that the issue keeping this server down was a bad mount to the file server.  We fixed this and the server was able to come back up finally!  This was done over 18 hours ago, so service should have been returned for a while now.  Of course, if you still experience any issues with this server, please contact Support for assistance.


2:12 pm

Central Equipment Move

Posted (November 18th, 2009 at 2:12 pm PST) by Patrick

Our final round of moves is occurring this Thursday (11/19) night beginning at 9pm pacific time and ending at 6am Friday morning. We will be moving all of our internal machines into new datacenter space. Most services on your account will remain up and functioning properly.

Central services that will be inaccessible during this time include the web panel, dreamhost.com, DreamBook, Files Forever, Easy One-Clicks, DreamHost Apps, media.dreamhost.com, wiki.dreamhost.com, discussion.dreamhost.com, basically everything but secondary DNS and this status blog.

So in summary, DreamHost systems will be unavailable, but all customer email, websites, DNS, and databases should remain up, available, and fully functional.

Please check this space for updates and thank you for your patience with all of these server moves.

— UPDATE Fri Nov 20 00:24:46 PST 2009 —

All machines are moved and most everything is back up and running. We’re at about 90%. You can access the panel again as well as most services. More information to come!

— Update Fri Nov 20 03:34 PST 2009 —

One of our central database machines that stores sessions had an issue and rebooted itself. Our MySQL admin is looking into the issue but it is causing panel downtime.

— UPDATE Fri Nov 20 04:04:55 PST 2009 —

4 o’clock and all’s welllll! Despite the brief hiccup we had with the database that stores our session information, we have been back up and functioning properly for several hours. If you have any problems with your service whatsoever, please contact support.

Again, thank you so much for your patience!

– The Happy DreamHost Server Moving Team


2:58 pm

Downtime on various Homie machines ( Continued )

Posted (November 17th, 2009 at 2:58 pm PST) by brians

We jumped the gun when announcing that the network problems were resolved. There is still latency and packet loss occurring on the machines. This would explain the degradation of service that you are experiencing. Our admins are diligently working towards getting this issue resolved. We plan to add more fiber to the network within the next few days. Again, we are really sorry for the problems this has caused. Your continued patience is greatly appreciated. However, your frustration is completely understandable. Thanks again for hanging in there with us. Please check back for future updates.

UPDATE 2009-11-19 9:24AM PST: Sorry for the delay in updating this post. We have upgraded our network for the areas that have needed it the most. We are optimistic that beefing up the network in these areas
will resolve the problem. Things are looking a lot better. Our network engineers will continue to actively monitor the stability of the network.

UPDATE 2009-11-19 9:42AM PST: Unfortunately, we are still noticing network hiccups. Our admins are still investigating the matter. Again, we sincerely apologize for the inconvenience this has caused. Please check for for updates.


1:22 pm

Email issues

Posted (November 17th, 2009 at 1:22 pm PST) by andrea

We’ve been experiencing email problems this morning, including inability to send or receive. Our admins are looking into it, and we’ll update this post as soon as there is progress in the matter. Very sorry about the inconvenience this is causing!

UPDATE 3:55pm PST – We believe this issue to be resolved as of about an hour ago. If you’re still having email problems, please let us know with details, so we can assist you.


10:48 am

Networking issues with MySQL servers

Posted (November 17th, 2009 at 10:48 am PST) by danwy

We’re currently experiencing some slow pings and packet loss to some of our MySQL servers in one of our server racks. We believe this is due to one of our links being saturated. Our networking team is working on adding more capacity to resolve these issues and hope to have things back to normal soon. We will update this status post when we have any more information regarding this issue.

Update – 8:30am (PST): Sorry for the delay in posting an update regarding this situation. It seems that things are a lot better now. Our network providers came out yesterday afternoon and added more bandwidth, which seemed to fix the immediate issues mentioned above. They are back this morning continuing to work on it and things do seem to be running a lot smoother already, so we’re hoping this is the end of this issue. We’re sorry for any inconvenience this caused you.


5:05 am

lovejoy getting new hardware after failure

Posted (November 17th, 2009 at 5:05 am PST) by Ken

Sad to say, lovejoy died and would not come back up.

Your admin team is in the process of giving lovejoy some new hardware to serve you even better.

We regret the inconvenience, and appreciate your patience while we upgrade your server.

Updates will be posted here as soon as we have more information for you.


7:32 pm

homie-vserver54 is toast

Posted (November 16th, 2009 at 7:32 pm PST) by Jason C

So we’ve got some bad news for the folks on homie-vserver54…  It would appear that the host server has decided to have a fairly glorious failure. One that requires brand new hardware and a full server restoration from backups.  This means that the following Private Server customers will see issues with their sites for the time being:

ps10028 ps10030 ps10039 ps10205 ps10208 ps10210 ps10265 ps10284 ps10343 ps10376 ps10415 ps10416 ps10417 ps10418 ps10644 ps10656 ps10657 ps10658 ps10660 ps10661 ps10664 ps10665 ps10684 ps10854 ps10911 ps10915 ps10943 ps11027 ps11028 ps11029 ps11130 ps11132 ps11133 ps11167 ps11173 ps11332 ps11334 ps11338 ps11557 ps12017 ps15082 ps15084 ps15234 ps15245 ps15246 ps15832 ps17792 ps17960 ps18194 ps9507 ps9508 ps9509 ps9513 ps9514 ps9516 ps9517 ps9520 ps9522 ps9525 ps9528 ps9543 ps9899 ps9906 ps9907 ps9908 ps9910 ps9912 ps9916 ps9919 ps9922 ps9929 ps9930 ps9942 ps9954

Our admin team is doing everything they can to make this restoration go quickly.  We’ll post updates here as things progress!


9:40 am

Downtime across various Homie machines

Posted (November 16th, 2009 at 9:40 am PST) by brians

We are experiencing packet loss on multiple machine. As a result, this is causing a lag when attempting to access your site. Our admins are currently working towards resolving the issue. Again, we are really sorry for all the recent problems. Your continued patience is greatly appreciated. Please check back for future updates.

UPDATE 2009-11-16 14:50PM PST: The swtich to our core router was oversaturated. Our administration team has added 2GB to the switch. We will continue to monitor the network. Sorry again for any inconvenience this has caused. Please check back for more updates.

UPDATE 2009-11-16 18:00PM PST: We have not noticed any more latency on the network. It appears to be under control. Our administration team will continue to actively monitor the stability of the network. Again, we sincerely apologize for the inconvenience this has caused. Thanks for hanging in there with us! However if you are still encountering stability issues, please submit a support request.


11:58 am

Private Server moves causing downtime

Posted (November 14th, 2009 at 11:58 am PST) by mir

Some customers are experiencing problems with their sites not loading properly. This is due to networking issues that have cropped up and our admin team is working hard to correct the problem. We’ll update this post as soon as this is resolved. We apologize for the inconvenience this has caused.

Update (2009-11-14 13:30 PST):

Our admin and dev teams are working together to resolve this and have hotfixed our move script so that we can correct the problem. We’ll update this post again when we have more information.

Update (2009-11-14 15:25 PST):

We were able to narrow down the issue (it was specific to customers with a unique IP that was not in the right part of our network to be moved to the new host machine) and have fixed it. We’re finishing up the configs that will get everything running so most of the effected customers are back up but we’ll update this again when those have all completed.

Update (2009-11-14 16:50 PST):

We believe this issue is fully resolved and have fixed the root cause so it will not occur again. We do apologize again for the inconvenience caused as well as the length of time required to fix this issue.


5:39 pm

Homie machine moves! 8pm tonight!

Posted (November 13th, 2009 at 5:39 pm PST) by Leon

The following machines are being moved from our old datacenter to our new datacenter. With any luck, downtime will be under 10-20 minutes per machine. The following machines will be moved:

angel apocalypse avalanche bishop blade bullseye cannonball constantine deathlok dredd exodus ghostrider greenlantern hawkman hobgoblin homie-vserver36 homie-vserver38 homie-vserver39 homie-vserver40 homie-vserver49 homie-vserver50 homie-vserver51 homie-vserver52 homie-vserver54 humantorch johnnyblaze luthor mrfantastic mystique namor preacher professorx proty quicksilver sabretooth sandman shadow silversurfer snowbird spawn spectre tick twoface wasp wonderwoman zatara

Update 11:27PM PST: All servers listed above are now back up and running in their new locations. If you’re still experiencing issues and your server is listed above, please contact support.


 
© 1996-2009, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
11:15 am

Networking problems across various homie machines

Posted (November 13th, 2009 at 11:15 am PST) by brians

We are experiencing a network outage at the moment. The following machines are affected. Our administration team is investigating the issue at the moment. We will post more details when they are available. Please check back for future updates.

altoid angel apocalypse avalanche bishop blade bullseye cannonball constantine deathlok dredd exodus ghostrider greenlantern hawkman hobgoblin humantorch johnnyblaze larchmont luthor mrfantastic mystique namor preacher professorx proty quicksilver sabretooth sandman shadow silversurfer snocap snowbird spawn tick twoface wasp wilshire wonderwoman zagnut zatara

UPDATE 2009-11-13 12:49PM PST: The network issue has been resolved. We decided to move the machines to another switch. We sincerely apologize for the inconvenience this has caused and appreciate your continued patience. If you are still having problems, please submit a support request.


10:31 am

Stability issues on Dillinger

Posted (November 13th, 2009 at 10:31 am PST) by brians

The machine is experiencing disk issues and failing to boot. Our admins are diligently working towards getting the machine back online. We sincerely apologize for the inconvenience this has caused and appreciate your continued patience. Please check back for future updates.

UPDATE 2009-11-13 16:07PM PST: We are in the process of moving the machine to new hardware. The machine be back online shortly. Again, we truly apologize for the downtime that you are experiencing. Thanks for hanging in there. Please check back for future updates.


8:09 am

Homie and Postal Cluster Networking Problems

Posted (November 13th, 2009 at 8:09 am PST) by Justin@DH

This morning we found some networking issues causing downtime with the postal cluster, where file server links hosting customer data were having delays, and for the homie cluster some servers were having issues to start normally. We found the networking problem and have added additional hardware to handle the higher than expected file server traffic. We are monitoring it further and cleaning up any servers that currently are having problems related to this. When our monitoring system shows things all fixed up, we will resolve this post. Our apologies for the downtime this morning.


3:11 am

frisky-vserver02 moving to new hardware

Posted (November 13th, 2009 at 3:11 am PST) by Ken

Due to a hardware issue, frisky-vserver02 is being moved to new hardware.

This post will be updated as soon as information becomes available.

UPDATE: guest private servers are bloghogwrts, brompton ecsos, learnwake, ps4622, radix, sindhu, unisoft, and valhalla.

UPDATE: Sorry for the delay in getting you restored. All guests are now back up and running. We regret the inconvenience but appreciate your patience while we upgrade our hardware.


8:00 pm

Moving Homie Machines Tonight!

Posted (November 12th, 2009 at 8:00 pm PST) by brettg

We will be movingHomie Cluster machines to a new physical location tonight. Similar to the last several moves we’ve ran, the amount of downtime should be approximately 10 minutes for your server. The move will begin at 20:00 PST (8 PM). Once we’ve completed the moves, we will update this status post. Thanks for your continued patience!

The servers that will be moved are:

jubilee, schultz, scalise, phantom, trafficante, homie-vserver02, homie-vserver04, homie-vserver05, homie-vserver06, mcgurn, rothstein, obelix, homie-vserver55, mangano, gotti, cloak, dagger, hawkeye, havok, cable, batman,capone, mothman, giancana, supergirl rogue wolfsbane sunspot warmachine forge thor starfire huntress haddock mekon cerebus swampthing marv,  bonanno bruno dalitz barker

— UPDATE Thu Nov 12 22:30:55 PST 2009 —

We got a bit of a late start (21:00 instead of 20:00) but all of the servers have been moved and are back up! If you have any questions or concerns at all, please contact support.

Thank you for your continued patience with all of these servers moves.

– Your Happy DreamHost Moving Robot Units.


6:30 pm

Mysql Moves Tonight! (November 12th 2009)

Posted (November 12th, 2009 at 6:30 pm PST) by brettg

We will be moving 22 MySQL Machines tomorrow at around 18:30(PST). The entire move shouldn’t take more than a couple of hours, with each machine experiencing significantly less downtime than 2 hours.

The machines that will be moved are:

yellowtail, homie-vsql10, homie-vsql11, homie-vsql12, homie-vsql14, homie-vsql15, homie-vsql16, homie-vsql17, spongebob, yogi, elmerfudd, shaggy, swordfish, cod, tuna, sardine, flounder, terrance, phillip, towelie, felix, fatalbert

If you have any questions, please contact our support staff.

Thanks!

Happy DreamHost Moving Squad!

Update 19:11PM November 11th: We’ve decided to postpone these MySQL moves, there is no time frame to move these MySQL machines. When we decide to move these machines, we will post an update similar to this.

Update 12:01PM November 12th: We will be going ahead with the move tonight at 6:30PM (PST). Again, these machines should not be down for longer than 10 minutes and we will update this post once the move has completed. If you have any questions, please contact our support staff.

Update 20:30PM November 12th: The moves have completed for these MySQL machines, if you notice further problems please contact our support staff.


6:00 pm

More Homie and MySQL moves!

Posted (November 10th, 2009 at 6:00 pm PST) by brettg

We will be moving some web servers belonging to the Homie cluster tonight along with a few MySQL machines over to new locations. MySQL moves will begin at 6pm (PST) and web servers will begin at 9pm (PST). Each moved machine should not be down longer than 10 minutes.

The MySQL machines that we will be moving are:

avarice, bang, domination, hazelnut, homie-vsql04, homie-vsql05, homie-vsql06, homie-vsql07, homie-vsql08, homie-vsql09, koi

The web server machines we will be moving are:

biggs, buchalter, burke, castiglia, cohen, dillinger, dio, escobar, gambit, greene, greengoblin, hoffa, homie-vserver07, homie-vserver08, homie-vserver09, homie-vserver10, homie-vserver11, homie-vserver12, homie-vserver13, homie-vserver14, homie-vserver33, homie-vserver34, homie-vserver35, homie-vserver46, ironman, karpis, kingpin, lepke, masseria, moran, provenzano, siegel, spilotro, storm

Thank you again for your continued patience, we will update you as soon as all of the machines have been moved and are live.

-Your Truly Happy DreamHost Moving Team!

Update 18:55 (PST): All MySQL machines have been moved to their new locations and are currently up and running. If you notice  problems with your MySQL services, please contact our support staff.

Update 11:20PM PST: All of the homie shared/vservers servers have been moved and now back up. If you’re still experiencing issues, please contact our support staff.


9:16 am

Homie Mail Downtime

Posted (November 10th, 2009 at 9:16 am PST) by Kyle

A few of our database clone servers had problems replicating from the master server this morning and caused some mail disruptions.  The problematic clones are no longer being queried by the mailservers for the time being to mitigate this issue.  If you are still experiencing problems please submit a support ticket from the webpanel!


8:09 pm

Database Server Moves

Posted (November 9th, 2009 at 8:09 pm PST) by Patrick

We’re moving more database servers tonight at 9pm PDT. The following machines will be down for roughly 10 minutes a piece:

hooch cujo bingo sekhmet sanspoof supercat mau bast solomon tom bigglesworth rhubarb nala bill ren puss bagheera milo sabu snowdrop sizzle arlene sifl

Check this space for updates!

— UPDATE Mon Nov 9 22:03:06 PST 2009 —

All machines have been moved and are back up and functioning properly. If you have any problems whatsoever please contact support.


3:03 pm

Homie MySQL Downtime

Posted (November 9th, 2009 at 3:03 pm PST) by Justin@DH

We are investigating a networking problem that is effecting some mysql servers in the homie cluster. Our networking admins are looking at it right now and we are working to get this backup was soon as possible.

Update – 3:56pm (PST): The service has been restored.. more details about the cause and resolution forthcoming!


 
© 1996-2009, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
12:26 pm

Moran server is being a moron

Posted (November 9th, 2009 at 12:26 pm PST) by danwy

The server ‘moran’ is having some networking issues and is causing some downtime for customers on that server. We’re investigating it now and may be replacing its network card. We’ll update this status post when we believe the issue is resolved.

Update – 12:51 pm (PST): Turns out that the server is actually the victim of a DDoS attack. Our admins are working on rerouting the attack traffic and getting things back to normal. Please keep an eye on this post, as we’ll update it when we have any more relevant information to this server.

Update – 1:40 pm (PST): This should be resolved now. We’ve re-routed the traffic from the DDoS and rebooted the server. We’re keeping an eye on it to make sure that things run smoothly. If you continue to have problems, please contact support.


12:02 pm

SquirrelMail Instability

Posted (November 9th, 2009 at 12:02 pm PST) by brians

We are fully aware of the delay, connections errors, etc. that you are encountering when accessing your mailboxes via Squirrel Mail. There are inflated loads on the webmail machines. Our administration team are diligently working towards resolving the issue. We sincerely apologize for the inconvenience this has caused and appreciate your continued patience. Please check back for future updates.

UPDATE 2009-11-10 13:39PM PST: Everything has been resolved. A few of the machines that handle the WebMail service required a reboot. Again, we sincerely apologize for the inconvenience this has caused. Our administration team will continue to actively monitor the stability of the machines. If you continue to encounter issues with SquirrelMail, please submit a support request.


2:12 pm

Even More MySQL Moves!

Posted (November 6th, 2009 at 2:12 pm PST) by micah

We will be moving more customer database servers Friday night, November 7, starting at 10PM PDT. We expect the move to take 3 hours, ending at 1AM PDT on Saturday. Each server will likely be down a fraction of that time.

The following servers are affected:

snowball garfield meow toonces luna guinness hobbes orion dandilo cassandra figaro mimsey rumple burbank aslan roary natty hooch cujo bingo sekhmet sanspoof sparky barbaro seabiscuit cykotix masevo fwap naroga petrelli bennett grendel lingling puff ein

We appreciate your continued patience as we work to streamline and reorganize our server layouts. Also, I apologize for the late notice of this maintenance window, I apparently didn’t notice that WordPress had eaten my previous post. If you have any questions regarding any of this please contact support.

Happy DreamHost Movin’ Team!

Update — Sat Nov 7 00:56:36 PST 2009 — All clear, the move is over and every MySQL server is responding normally.


12:18 pm

Jasper is offline

Posted (November 6th, 2009 at 12:18 pm PST) by brians

The machine is not responding to the remote reboots that are being issued. A member of our administration team is currently investigating this matter. The machine should be back online shortly. Unfortunately, I cannot provide an exact ETA. We truly apologize for the inconvenience this has caused. Please check back for future updates.

UPDATE 2009-11-06 12:24PM PST: Jasper is back online. It just needed a manual reboot. We will continue to monitor the stability of the machine. Again, we sincerely apologize for the inconvenience this has caused and appreciate your continued patience. If you are still having problems, please submit a support request.


9:04 am

PS host instability issues.

Posted (November 6th, 2009 at 9:04 am PST) by mir

We have a couple of hosts that are running into kernel issues which is causing downtime for Private Server guests on those hosts. We’re working with the VPS development team to generate a solution that does not undermine security on those machines. We’ll update this post as soon as we have more information.

Update (2009/11/06):

We’ve got a patch that should let us isolate and correct the problem we’re having – it’s not as simple as rolling back the kernel change due to a vulnerability with the previous version (it wouldn’t be secure to just revert) so we have to get this fixed. We’re doing our best to get it corrected as quickly as possible.


4:33 am

Pistone server down

Posted (November 6th, 2009 at 4:33 am PST) by patrickm

We are currently investigating an issue with the Pistone server, which is causing downtime.  We apologize for the inconvenience and hope to have the issue resolved shortly.  Please check here for updates.

UPDATE 1:20pm PST – We are getting ready to move Pistone’s data to new hardware. Sorry about the continued inconvenience, we’ll update this post again with any progress made.

UPDATE 4:10pm PST – Unfortunately, the first replacement machine we tried didn’t work out. We’re getting another one ready for Pistone, and will attempt the restore from backups as soon as the new hardware is ready.

UPDATE 6:25pm PST – The new machine is up, and all apache web services are running. We’re looking into the current high load, so this post will not be resolved until we made sure the server is running smoothly. Thanks for your patience!

UPDATE 2009/11/07 10:30am PST – We have noticed the load spiking on the server and are investigating it to see what we can do to properly stabilize it.

UPDATE 2009/11/07 1:25pm PST – We’re monitoring the server to try to keep it under control and are looking to see if we can find any user processes that are contributing to the high loads. It does look like there will be some instability until the data copy to local disk is complete but we should be able to keep the server up and running during that period (we may have to restart the apaches occasionally if stale processes pile up though).


10:00 pm

Homie Mail file server move 2009-11-05 10pm pacific

Posted (November 5th, 2009 at 10:00 pm PST) by kitchen

As part of our rack consolidation efforts we will be moving two four of the file servers which provide storage for homie mail on Wednesday 2009-11-04 2009-11-05 at 10pm pacific. This will result in some downtime for customers on those file servers. The file servers we’re moving are milesaway, fromordinary, tastesgreat, and lessfilling. The estimated downtime for this move is approximately 2 3 hours, during which time your incoming mail will simply queue up on our mail servers waiting to be delivered.

We thank you for your patience and apologize in advance for any inconvenience this downtime may cause.

Update — Wed Nov 4 18:00:53 PST 2009, we have postponed this move until tomorrow night. The post date reflects the new maintenance window 2009-11-05 22:00:00 – 2009-11-06 00:00:00. We apologize for the short notice change in plans.

Update — Thu Nov 5 22:02:58 PST 2009, we are beginning the moves now. We will post an update here once everything is finished! Thank you for your patience and we apologize for any inconvenience.

Update — Fri Nov 6 01:20:48 PST 2009 Everything is back up and running! If you are still having any issues, please contact support!

Update — Fri Nov 6 02:06 PST 2009 The moved mail servers are back up, but an issue with our authentication servers was just identified.  We’re working on a resolution now and will update this post when everything is working again.  Thank you for your patience during this time.

Update — Fri Nov 6 07:11 PST 2009 – The lingering authentication issues are now resolved.  We truly apologize for the inconvenience.  If you are still having issues with any of your hosted email accounts, please contact Support and they will be very happy to assist.

Update — Fri Nov 6 8:00 PST 2009 – We sincerely apologize, as there is still an issue with one of the mail servers in the Homie cluster that is causing email/webmail downtime.  We are resolving this issue as I type this, so we expect it to be resolved shortly.  Again, our deepest apologies about this continuing issue.

Update — Fri Nov 6 11:20 PST 2009 – We believe we have identified the problem, and are working on the resolution. We’ll update this post again with any progress made.

Update — Fri Nov 6 1pm PST 2009 – The problem causing authentication, sending and receiving issues on your mail servers are now resolved. We’ve been testing numerous accounts in the past hour to make sure, and everything appears to be working again. You may experience short delays – connections may clog up a bit initially due to the rush of sending and receiving after the services were restored. If you’re waiting for mail that was sent to you during the data migration, those are being queued either on the sender’s server or on our incoming mail servers, and should be delivered shortly – unless they bounced back to the sender immediately after the first try. Sorry again about the inconvenience this mail downtime has caused. If you’re still having problems with your mail, please submit a support ticket with details, so we can look into it for you.


2:19 pm

Yankees cannot buy uptime!

Posted (November 5th, 2009 at 2:19 pm PST) by mir

We apologize for the issues customers on the ‘yankees’ server have been experiencing. The problem was due to hardware (the RAID card to be specific) but we have that corrected and are bringing it back up now (it’s just taking a bit longer than normal as we’ve had to go through the post deployment process). It should be available in the next 20-30 minutes but we’ll update this post once we have it back up again.

Update (15:45 PST):

Yankees is back up and running and should be fine but the admin team is keeping an eye on it just in case.

Update (2009/11/06 8:15 PST):

Yankees is back up again and we’re keeping an eye on it to make sure it remains stable. Basically the raid card was having issues with one of the defined devices which meant we needed to power cycle it but it looks all better.

Update (2009/11/06 10:05 PST):

The issue is recurring so the admin team is looking into whether or not we need to move the machine to new hardware – will update this post once we have determined the best course of action.

Update (2009/11/06 13:00 PST):

It is up currently but it refuses to remain stable so we’re in the process of replacing the hardware entirely (the admin team just needs to put together a replacement machine as we didn’t have one ready to go like we usually do). They’re going to get that taken care of as quickly as possible and we’ll update the post here as soon as it happens

Update (2009/11/06 16:10 PST):

Yankees is back up on new hardware and will be stable – for now we have mounted the data from our backup server and are copying it over to the local disk which may effect performance slightly until it completes but the server should be back up for good. We apologize again for the inconvenience due to the hardware failure.

Update (2009/11/07 10:30 PST):

It looks like the new server is getting overloaded with sleep state processes (driving up the server load). We were able to get it back down under 20 and will be keeping an eye on things to try to manage it as well as we possibly can (please bear in mind that we’re still copying the data from the backup server to the local disk which will contribute slightly to the overall load and thus magnify activities that bring the load up).

Update (2009/11/07 13:20 PST):

We’re monitoring the server to try to keep it under control and are looking to see if we can find any user processes that are contributing to the high loads. It does look like there will be some instability until the data copy to local disk is complete but we should be able to keep the server up and running during that period (we may have to restart the apaches occasionally if stale processes pile up though).


11:04 am

Bulger is unhappy

Posted (November 5th, 2009 at 11:04 am PST) by Terri

We are in the process of moving everyone on bulger to new hardware. The filesystem kept mounting read-only, which continually was making things unstable for the customers using the machine.

We’ll try to make this as quick and painless as possible, but it will take some time. Please watch this space for updates.

This only affects the web server. Customers on bulger will have access to their email and databases.

Update 5:20 P.M. PST: We are still restoring data on bulger. Once data is restored from backup we will do a final sync with the data from the old machine.


6:16 pm

Multiple Server Reorganizing

Posted (November 4th, 2009 at 6:16 pm PST) by micah

Tonight we’ll be moving 26 of our web servers from one rack to another. The destination rack is fully prepped so downtime shouldn’t be more than 10-20 minutes per server. We will begin at 10pm PST and will update this post with the all clear.

Sorry for the last minute post, we’re having to shuffle quite a few things we hadn’t planned on.

Here’s the list of affected machines:

bitters krabappel clancy arnie ned dawber chalmers bart shemp salt manjula abe brockman gimli mona patty jebediah brawndo jimbo quimby duffman drederick jasper mccallister

Update 11:20pm pacific: And these are all done! If you are still experiencing any issues, please contact support!


 
© 1996-2009, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
5:06 pm

fscking bulger

Posted (November 4th, 2009 at 5:06 pm PST) by kitchen

The machine ‘bulger’ is having some problems with its filesystem and we’ve taken it down to run an fsck on it. We will post updates here as things progress.

Update Nov 5th, 11:30am (PST): The file system check on this server has completed. We are now in the process of moving this server to new hardware. Please keep updated at this new post:

http://www.dreamhoststatus.com/2009/11/05/bulger-is-unhappy/


4:23 pm

Homie downtime across various machines

Posted (November 4th, 2009 at 4:23 pm PST) by brians

We had to upgrade the kernel on several of the homie machines. A reboot is required upon completion. However, a few machines are still having problems. We are looking into those machines now. The machine whose kernel did not seamlessly upgrade will be reverted back to the previous kernel version. Here is a list of machines that we have attempted to upgrade. We hope to have service restored ASAP. Sorry for the inconvenience this has caused. Your patience is truly appreciated. Please check back for future updates.

addisababa alouettes andross angels argonauts asmara avengers baku bandar bayhawks bears beijing belmopan bengals biggs bills blackhawks blazers bluebombers bluejays bobcats bogota brasilia brazzaville brewers bridgetown broncos browns bruins brumbies brussels buccaneers buchalter bucks buenosaires bujumbura bulls burke cairo canberra cannons cardinals castiglia cavs chivas clippers cohen colossus colts coonan copenhagen cowboys crew crusaders cyclops d-fenders dedede dhaka diamondbacks dili dio djibouti dodgers dynamo eagles earthquakes ecco escobar eskimos expos falco falcons fever fire fitzpatrick flames force gaborone galactus galaxy gambino genovese ghostrider giancana gotti gravano greene greengoblin grizzlies havana havok hawkeye hawkman hawks heat herediano highlanders hobgoblin hoffa hornets hulk hurricanes iceman indians islanders jaguars jazz jets joker jubilee kabul kakuji karpis kings kinshasa knicks lakers lansky lapaz legs lepke liberty lightning lions lizards luanda luthor magic magneto malabo manama mangano mariners marlins masseria mcgurn mclean mets monarchs moran moroni mrfantastic nassau nationals ndjamena nets nicosia nightwing niners nitti nuggets obelix oilers olympiacos ottawa ouagadougou outlaws pacers packers padres panthers patriots penguin penguins peppy phillies phnompenh pirates pistons porto-novo prague praia preacher predators pride provenzano quicksilver quito rams rangers rapids raptors ravens rays reds redwings revolution robin rockets rockies roseau rothstein roughriders royals sanjose sansalvador santiago santodomingo saprissa sarajevo schultz seahawks shadow shock silversurfer slippy snowbird sofia sox sparks spiderman spillane spilotro spurs stampeders stjohns sugeknight suns suva tallinn terranova texans thimphu thing thrashers thunder tick tigercats tigers tirane titans twins venom vikings walsh waratahs wario warriors wasp watanabe wizards yamoussoukro yankees yaounde yerevan zagreb zwerbach

UPDATE 12:31AM Pacific: This actually also applies to a number of servers under other clusters.  Some of the servers under Blingy, Postal, Randy, Frisky and Spacey were also updated with the new kernel, which will have caused some downtime as we rebooted the servers.  We still have a small number of servers that need this update, but we are done with the updates for the night.

If you are having any issues with your hosting, please don’t hesitate to contact Support for assistance.  A new post will be added for the updates that happen tomorrow, so please check back for that.


10:00 pm

More Database Server Moves

Posted (November 3rd, 2009 at 10:00 pm PST) by DreamHost

We will be moving more customer database servers Tuesday night, November 3, starting at 10PM PDT. We expect the move to take 3 hours, ending at 1AM PDT on Wednesday. Each server will likely be down a fraction of that time.

The following servers are affected:

sparky sanspoof morris fin flipper cujo buck cheshire tigger sekhmet hooch bingo jarvis hushpad catbert lionel cinnamon growltiger bastet jake moglie snarf leo midnight neko heathcliff jones scratchy demeter sarabi elmyra drooper jingoro ragland streaky tammany skimbleshanks tachibana

–UPDATE We’ve added a few more servers to the list, please read over them to be certain whether you are affected –Tue Nov 3 16:21:34 PST 2009

–UPDATE All servers are back online. –Tue Nov 3 23:55:00 PST 2009


11:05 am

Private server issues

Posted (November 3rd, 2009 at 11:05 am PST) by andrea

Currently, one of our private server hosts is running a file check to confirm data integrity on that machine. The affected virtual private servers are

ps12644 ps12652 ps12657 ps12663 ps12672 ps12693 ps12697 ps12706 ps12709 ps12712 ps12715 ps12718 ps12720 ps12723 ps12730 ps12733 ps12737 ps12741 ps12745 ps12750 ps12759 ps12767 ps12772 ps12774 ps12779 ps12799 ps12803 ps12807 ps12821 ps12827 ps12839 ps12847 ps12856 ps12882 ps12894 ps13014 ps13015 ps14196

We apologize for the inconvenience, and ask you to check back here for updates. We don’t estimate this to take too long.

UPDATE 12:00pm PDT: We’re happy to report that the server is back up. If you continue having issues with your service, it is most likely unrelated to this outage. Please submit a ticket with the details, and we’ll look into it for you.


10:00 pm

Database Server Moves

Posted (November 2nd, 2009 at 10:00 pm PST) by DreamHost

We will be moving some customer database servers next Monday night, November 2, starting at 10PM PDT. We expect the move to take 3 hours, ending at 1AM PDT on Tuesday. Each server will likely be down a fraction of that time.

The following servers are affected:

amber argos barfolomew cerberus clifford courage duchess duke fang fluffy garm gibson greebo hercules hushpuppy joecool juno kona krypto lassie menchit monty mufasa muffy odie putney sawdust scarlet scooby sebastian simba sirius skip snagglepuss spike spot sprocket spunky-vsql02 spunky-vsql03 spunky-vsql05 spunky-vsql06 sui tiny tomoe tramp ubu underdog yeller

UPDATE — We are beginning these moves in 5 minutes –Mon Nov 2 21:58:11 PST 2009

UPDATE — We finished and all MySQL servers were back up by –Mon Nov 2 23:30:00 PST 2009

If you continue to have problems with your database on one of the above servers, please contact support.


10:18 am

Hanjin backup server down.

Posted (October 30th, 2009 at 10:18 am PST) by mir

We are investigating the cause now – we moved the server last night in an effort to set it up for a permanent resolution to the issues that have been plaguing it but now it’s not working at all. The admin team is trying to fix this and will provide details as they become available.

Update: 3:09 pm pacific –
Currently, hanjin is still down. We have decided to take it offline for the time being as there is a defective hard drive in one of its arrays which is causing excessive slowness on the machine. Additionally, we feel that the machine’s overall load is also affecting the rebuild status, so they’re fighting each other and it’s not going well. We expect to have an update on this situation within the next few hours, but the drive rebuild will definitely take some time. Thank you for your patience and we sincerely apologize for any inconvenience this has caused.

Update: 10:33am pacific (monday) –
We let the drive try to rebuild over the weekend and unfortunately it hasn’t finished rebuilding. However, we have changed a few options which should hopefully increase performance and prevent the system from becoming completely unresponsive during the rebuild process and have turned hanjin back on this morning. We will continue to monitor the situation and provide updates as things progress.

Update: 16:58 pacific (tuesday) –
The admin team has informed me that this machine is down again but they are aware of it and working on a resolution. We’ll update this again once we have a handle on the current situation.

Update: 15:20 pacific (sunday) –
The drives on the backup machine are still rebuilding. Our current ETA for this to complete is tomorrow. Hopefully when this is done it will remain stable enough for us to work on it and get it offloaded to an additional server to free up some resources.

UPDATE 2009-11-13 13:08PM PST: The drive rebuild reach 98%, then stopped. We had to initiate the rebuild process from the beginning. It should complete within the next 48 hours. Sorry for the inconvenience this has caused. Your continued patience is greatly appreciated.

UPDATE 2009-11-19 10:47AM PST: The rebuild is in the home stretch now. Only about 3% remaining to be completed. If after this is completed things don’t improve dramatically we may have to take more drastic measures to improve the performance of this service.

UPDATE 2009-11-24 8:00AM PST: Before the last rebuild could complete another drive failed. This means that there is another rebuild in process now. “When it rains it pours.” Unfortunately I don’t have an ETA yet on when we expect this current rebuild to finish. Thanks for your continued patience and understanding while we’ve tried to resolve this issue.

UPDATE 2009-12-02 12PM PST: The rebuild is now finished. If you’re still having issues with your backups, please submit a support ticket with the details. Sorry again about the inconvenience this long process has caused.


8:11 am

Dowtime across multiple machines

Posted (October 30th, 2009 at 8:11 am PST) by mir

We have been experiencing a networking issue with an isolated cluster of machines in which they would lose connection to their NFS file system (which experts their root file system). The admin team is still investigating but these are the effected machines:

hornets rockies nicosia prague djibouti roseau santodomingo dili quito cairo sansalvador malabo asmara tallinn addisababa suva homie-vserver101 homie-vserver102 homie-vserver103 homie-vserver104 homie-vserver105 homie-vserver106 moroni brazzaville kinshasa sanjose yamoussoukro zagreb havana homie-vserver115 joker sugeknight banshee eastman homie-vserver32 penguin terranova floyd pistone brasco walsh zwerbach coonan fitzpatrick kakuji cummiskey spillane mclean wolverine thing legs riddler magneto hulk aquaman galactus venom spiderman cyclops iceman nightwing colossus catwoman nitti bugsy robin bulger castellano genovese homie-vserver113 copenhagen homie-vserver107 homie-vserver111 homie-vserver112 homie-vserver108 dodgers raiders kings galaxy clippers rams sparks blazers angels padres niners seahawks magic mariners warriors mets redwings knicks suns diamondbacks broncos tigers steelers lions bruins titans braves patriots

We’ll update this post once we have more information.

UPDATE 2009-10-30 08:58 PST All servers in this posting have returned to normal functioning. The admin team is still investigating the issue so that we can ensure that it does not crop up again.

UPDATE 2009-10-30 011:25 PST We believe that the issue was due to network saturation and/or a failed interface on one of our distribution switches.  The problematic interface has been turned down and we have directed some other traffic elsewhere with hopes that it will remedy the disconnects.

UPDATE 2009-10-30 14:24 PST At around 12:30 we replaced a gigabit interface converter and so far the network has stabilized. There are a few machines that are still having lingering problems, but the network side of this has been resolved. Sorry about the problems this has caused. If you have a support message in we are answering them now after verifying that your account is up and working properly again.


7:23 am

Gremlin experiences irony (mysterious harware issues)

Posted (October 30th, 2009 at 7:23 am PST) by mir

It looks like there is a problem with either the motherboard or power source on Gremlin. Our admin team is moving everything to new hardware which should be completed in the next 45 minutes or so. We’ll update this status post when that has been completed.

UPDATE 2009-10-30 09:15 PST Gremlin is back up and running so everything should be back in order but the admin team will keep an eye on the new hardware just to be safe.


10:00 pm

Server moves Thursday 29 Oct beginning around midnight

Posted (October 29th, 2009 at 10:00 pm PST) by Ken

To better serve you, we are in the process of consolidating racks so we can add more equipment. The outage for each server should be no more than about 10 to 15 minutes for the actual move.

Servers affected are:

carrie coolblue fxdollar godfather  jove limbo-randy montecristo smithers

PS’s that are affected are:

accserver browser celtics ed engine57 gassi1 gnif gobehost hyperblocks iandi iceandrock jtse keei maya mtrends oden phan popehat ps3710 ps3845 ps3900 ps3927 ps3957 ps4087 ps4326 ps4337 ps4473 ps4731 ps5037 ps5741 ps5752 ps6300 ps6428 ps6432 ps6467 ps6803 ps7283 ps7524 ps7922 softwa ssiws1  touchofdeathforums vpg wittenberg wyld xgun

We will update this post after servers have been moved. Again, the outage for each server should last no more than 10-15 minutes per server.

UPDATE 2009-10-30 02:34 PST: Server moves have begun.

Before reporting an issue, check if your server is on the list – if it is, check again after about 15 minutes to emsure that there might be a problem before reporting an issue please.

UPDATE 2009-10-30 07:45 PST: All affected servers have been noved.

If you still experience problems with your service, please follow the normal issues procedures, and someone will be happy to assist you.


10:00 pm

Hanjin going down for maintenance

Posted (October 28th, 2009 at 10:00 pm PST) by kitchen

We’re going to be taking hanjin (backup.dreamhost.com) offline this evening (2009-10-28) around 10pm pacific and moving it to another rack. The rack it is currently in is full, so we can’t add more hardware to alleviate some of the load issues it’s been having, and we need to move it eventually anyways. This is the first step in the process we’ll be taking which will hopefully take care of some of the instability of this service recently.

Estimated downtime is approximately 2 hours, and we’ll post updates back here once things are back up and running!

Update: 9:56pm pacific: hanjin is going offline now. We’ll update this post as soon as everything’s back up and running!

Update: 10:59pm pacific: hanjin is back up and running! We’ll be doing more to address the issues it’s been having over the next several days, thank you for your patience!


 
© 1996-2009, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
1:18 pm

A quick note on backups of backups

Posted (October 27th, 2009 at 1:18 pm PST) by kitchen

As you all should know, we do not allow the space on our servers (with the exception of our machines specifically online for this purpose) to be used to house backups. This policy has been in place for quite some time, and is nothing new.

What *is* new, however, is that we are no longer going to be making backups of these directories. Anything our backup system thinks is backups will not be backed up. How does this affect you? If you have these backups in your home directory, and we experience a loss of a filesystem and have to restore the machine from backups, your self-made backups stored on our servers (again with the exception of the machines hosting backup.dreamhost.com) will be gone. This is to save us considerable space on our backup storage systems, as well as saving considerable amounts of system resources when performing the backups of this data.

If you are currently using your shared storage space for backups, we recommend that you move these off of the system and either over to backup.dreamhost.com (where you get 50GB of free storage for backups) or to another service. If you do move it to another service, please make sure they allow using your storage space for backups or they’re a dedicated backup storage provider.

If you have any questions or concerns about this change, please feel free to contact support!

Update:

@Folks who are asking what the backup system thinks are backups:

Basically we won’t be backing up anything in directories named like they might contain backups, such as:
backups/
backup/
bkups/
bkup/

We will likely be adding additional patterns such as ones for rsnapshot-style backups and whatnot, but please believe me when I say we’re only adding the very blatantly obvious patterns, and not inspecting file contents or anything of the like to attempt to determine the backup status of files.


5:06 am

bulger going through drive cleanup

Posted (October 26th, 2009 at 5:06 am PST) by Ken

Webserver bulger is  going through a filesystem check. We will be bringing it back up as soon as it’s cleaned up. Sorry for the inconvenience and we’ll be bringing it up as soon as possible.

Update 8:00PST: The check finished about 30 minutes ago and service has been restored, if you are still experiencing problems please contact support!

UPDATE 2009-10-26 10:26PM PST: The filesystem issues have resurfaced. We are re-running the filesystem check. Unfortunately, we cannot provide you with an ETA. Again, we sincerely apologize for the inconvenience this has caused and appreciate your continued patience. Please check back for future updates.

UPDATE 2009-10-27 04:58 PST: bulger has finished it’s drive cleanup and checks and is now back online. Again, we apologize for the inconvenience and appreciate your patience.


4:46 pm

Copenhagen’s been chewed up

Posted (October 25th, 2009 at 4:46 pm PST) by Jason C

Due to a hardware failure on “copenhagen”, our admin team is in the process of migrating the server to new hardware.  It hopefully shouldn’t take too long to move everything over — but we’ll be posting updates here as the issue progresses.

Update – Oct. 25th, 5:24 PM (PST): Due to the state of the server, fixing everything up is going to take a little longer than normal.  So we were kind of dead wrong on our initial estimate.  Sorry about that!  We’re doing our best to get this resolved as fast as we can, however.

Update – Oct. 25th, 6:27 PM (PST): As of this moment, we’ve decided to go ahead and mount your data from our most recent backups. During this time we will be restoring your content from our backups to the new hardware.

Update – Oct. 26th, 7:10 PM (PST): Due to the filesystem flipping itself into “read only” status earlier today, the restore is taking even longer than anticipated.  It is still running and all content is expected to be restored eventually — but please note that the server will be running at reduced capacity (see: slower) until everything is back to normal.

Update – Oct. 26th, 9:25 PM (PST): One of our admins has restarted the Apache services on the server to help knock the load down.  This should stabilize the sites that are up and running for the time being.  Basically, the sites that are still missing are causing some Apache services to hang.  That ends up driving the load up.

One of our techs who has experience with recovering servers that have seen problems like this will be taking a look at things shortly.  Once we know where the recovery effort stands, we’ll make sure to update this article with more info.

Update Oct. 27th 12:32 AM PDT: I was able to get the array back into a workable state with no visible corruption. I started copying the data over to the new copenhagen now. An ETA before all data is restored is probably around 24 hours. Any data you upload that was newer than the original should not be over-written.

Update Oct. 27th 4:02 PM (PST): The data is still copying and on target to be finished in the (hopefully early) AM hours of the 28th.  Once the copy has completed, we’ll drop another update here.

Update Oct. 27th 10:50 PM PDT: All of the users we didn’t originally have backup data for have been restored from the original hardware except the largest 16 users (so the vast majority have been recovered). Users have been recovered in the order of disk space usage (smallest first) so all the remains are the largest users on the server. If you are on copenhagen, have less than 10GB of data in your users directory and you are missing files please contact support. This will be listed as completely resolved once these final 16 users are copied.

Update Oct. 28th 10:28 PM PDT: For several hours now all the data that we were missing backups from recent moves to the server have been restored from the original hardware. If anyone else is missing data please contact support to have your user manually restored.


2:23 am

Some PS’s down with from file-system errors

Posted (October 24th, 2009 at 2:23 am PST) by Sandon

Some of the PS web-servers are down due to some file-system errors which need an fsck to be run. This is affecting you if you are on one of the following PS’s:

ps15318, ps15329, ps15334, ps15345, ps15350, ps15356, ps15361, ps15366, ps15371, ps15377, ps15382, ps15387, ps15392, ps15397, ps15402, ps15407, ps15411, ps15427, ps15437, ps15461, ps15555, ps15686, ps15688, ps15689, ps15690, ps15697, ps15699, ps15700, ps15702, ps15712, ps15720, ps15721, ps15753, ps15777, ps15778, ps15780, ps15784, ps15804, ps15806, ps15810, ps15836, ps15837, ps15854, ps15864, ps15871, ps15878, ps15884, ps15896, ps15901, ps15906, ps15910, ps15918, ps15923, ps15928, ps15941, ps15946, ps15951, ps15956, ps15962, ps15969, ps15978, ps15983, ps15989, ps15997, ps16009, ps16022, ps16027, ps16032, ps16033, ps16078, ps16082, ps16090, ps16095, ps16098, ps16101, ps16103, ps16106, ps16108, ps16112, ps16113, ps16122, ps16123, ps16132, ps16134, ps16140, ps16144, ps16150, ps16151, ps16157, ps16158, ps16161, ps16163, ps16169, ps16172, ps16178, ps16181, ps16185, ps16194, ps16199, ps16202, ps16210, ps16212, ps16304, ps16362, ps16384, ps16388, ps16389, ps16395, ps16398, ps16410, ps16419, ps16429, ps16442, ps16445, ps16457, ps16460, ps16462, ps16466, ps16470, ps16471, ps16481, ps16482, ps16486, ps16487, ps16491, ps16492, ps16593, ps16641, ps16681, ps16712, ps16739, ps16762, ps16774, ps16806, ps16820, ps16822, ps16839, ps16861, ps16867, ps16902

There isn’t a ton of data on the file-system so it hopefully should be less than an hour for the fsck but these have been known to take a couple hours in some situations.

Update 03:02 AM PDT The first pass of the fsck has completed which usually means it is over half way complete so hopefully it won’t be much longer now.

Update 04:33 AM PDT Ok, the file-system check is complete and all the guests are back up. If you are still having any problems accessing your sites and you are on one of the above PS’s please contact support.

Update 08:42 AM PDT We already had made plans to do so but the machine failed again indicative of some problems with its raid controller. Due to this we have upped the priority on replacing it and should be taking the machine down in 1-1.5 hours (so probably around 10 AM PDT) to replace the raid controller.

Update 10:49 AM PDT We are taking the machine down now to replace the raid controller and will follow up here once that is completed.

Update 11:50 PM PDT: We have replaced the raid controller and the server is coming back up. All guests are coming back up without any issues.


10:35 am

File server Baster unhappy

Posted (October 23rd, 2009 at 10:35 am PST) by micah

One of our file servers has been having issues since around 7am PDT. We’re still working on getting it up and running and apologize for the downtime it’s causing customers on these machines:

alexander centipede contra cruisin donkeykong euro ikari ike io klax kroner limbo-randy mark morgan orbital pizarro qbert rupee

Web, SSH, FTP, and Jabber will either be slow or inaccessible in the meantime while the web servers are unable to mount the file server. Some customer’s mail will be inaccessible as well, no mail should be lost however, delivery should simply be delayed.

Our status page was having issues this morning as well, and we apologize also for the lateness of this post.

Happy DreamHost Friday Morning Funtimes Team!

UPDATE 2009-10-23 16:51 PM PST: This issue was resolved a couple of hours ago. Our administration had to replace a drive on one of the network fileservers that your machine is mounted to. They will continue to actively monitor the stability of the fileserver. We sincerely apologize for the inconvenience this has caused and appreciate your continued patience. If you are still encountering downtime, please submit a support request.


4:14 pm

Bulger crashed

Posted (October 18th, 2009 at 4:14 pm PST) by andrea

Our admins are working on getting our Bulger web server up again. Sorry about the inconvenience, and please check back here for updates.

UPDATE 5pm PST: We’re moving the machine to new hardware, so it will be a little while longer. Sorry about the continued inconvenience.

UPDATE 7:25pm PST: The server was successfully migrated to new hardware, and services have been reconfigured. If you’re still having issues with your domain, please don’t hesitate to contact our support team for assistance.


11:05 am

Greenlantern Lost His Ring

Posted (October 18th, 2009 at 11:05 am PST) by brettg

Server greenlantern has been experiencing issues with it’s RAID card, so we will be moving this server over to new hardware shortly. We apologize for the inconvenience, we will have an update as soon as the server has been moved to new hardware.

Update: Sorry for the lack of updates, it looks like the machine we had designed to move greenlantern too was also not willing to want to cooperate with us. We’re in the process of trying the move once more, we apologize for the delay!

Update: Sorry for the delay, again. There were some issues trying to restore the data on the drives. Unfortunately it looks like the data was corrupted, so we are restoring from our most recent backups of your data. Most users shouldn’t notice any data loss, if you do indeed notice the loss please contact our support staff. Unfortunately there is no time table for the restoration as it has to restore from our backup servers.

Update – 4:00pm (PST): We’re still in the process of restoring data from the backup server. In the meantime though, we’ve temporarily mounted the backup server over NFS so that everything should work correctly in the meantime. The reason that some sites were missing files is because the link to the backup server was broken and it wasn’t able to find them locally. They are still on our backup servers though and are in the process of being restored. We apologize for the inconvenience this has caused you. We will resolve this status post once the backups have completed and things are running normally.

Update – 10/25 5:30pm (PST): All restores are now done. However, if you’re missing files, please contact support with the details, so we can look into it for you. Thank you for your patience!


11:58 pm

Winder file-server down

Posted (October 14th, 2009 at 11:58 pm PST) by Sandon

It looks like one of our file-servers that some of or ‘spunky’ cluster servers use (Winder) is down for the count. One of our on-site staff is looking into the cause at this moment. This is causing some customers sites not to load and high loads on the following machines:

abbazaba absinthe achilles admiralty aero atlantic babyruth bandini baseline betsybird bigeasy bonbon bourbon bubbleyum bundy butler butterfinger centinela century cheeto cheezit cienega cognac cordial crenshaw dorito drumstick edendale euclid ewal fezzik foothill formosa grand granola grappa haven highland hillmont hoho hops jalapa jefferson kiki larchmont lemonhead limbo-spunky4 malt manchester mayo melrose modelworks monroe nash necco nougat nutrageous oakdale ocean ord outside payday pernod piggy pringle rexford rizzo roscoe rum sanborn schnapps scotch selma skittle skybar smores snocap soto sunset tequila toblerone truffle twinkie twix vcheck vermouth vine virgil warhead whopper wilshire zagnut zuma

You can see if you are on this file-server from the “Account Stuatus” page on the right hand of the panel if your file-server is listed as ‘Winder.’ More updates to come.

Update 12:35 AM PDT: It looks like one of the pieces of hardware the file-servers use connecting the disk-shelf to the head unit was bad and needed to be replaced. The file-server is now back up and running serving customer data and all services have been restored. If you are still having problems and are on spunky and this file-server please contact support.


5:59 pm

Server Moves

Posted (October 14th, 2009 at 5:59 pm PST) by Patrick

Tonight, starting at 8:30pm we will be moving some servers across the hall and between datacenters. Downtime for each server should not exceed 10 minutes. Here is a list of affected servers:

brasco bugsy bulger catwoman colossus coonan cummiskey eastman fitzpatrick floyd genovese gravano homie-vserver15 homie-vserver16 homie-vserver17 homie-vserver18 homie-vserver24 homie-vserver25 homie-vserver29 homie-vserver30 homie-vserver31 homie-vserver32 homie-vserver53 homie-vserver56 kakuji legs mclean nitti pistone robin spillane terranova thing walsh wolverine zwerbach

Update 11:05PM PST: Everything came back up without a hitch! We apologize for the lack of updates regarding the matter but we wanted to make sure everything was groovy before anything. If you’re experiencing any issues and your server is listed above, please contact our friendly support staff!

– The Happy DreamHost Server Moving Robot


11:33 am

MySQL server Phil is down

Posted (October 14th, 2009 at 11:33 am PST) by patrickm

We apologize for the inconvenience, as the MySQL server “phil” is currently inaccessible.  We suspect that the hardware might be the cause and we have dispatched an engineer to the datacenter to check it out.  We will update this post with more info as it becomes available.

Update 12:51: Phil’s been moved to new hardware and back up now! If you have any further issues with this machine, please contact your friendly DreamHost support representative!


 
© 1996-2009, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
10:37 am

Wario is out of extra lives

Posted (October 14th, 2009 at 10:37 am PST) by patrickm

The Wario server’s RAID card is experiencing some issues that are keeping this server offline at this time.  We sincerely apologize for the inconvenience that this is causing.  We’re working to restore functionality to this server now and will update this post when that is completed.

Update 12:19: Wario is back up and running. If you have any further issues with this machine, please contact your only-too-happy-to-help DreamHost technical support person!


9:50 am

Junk filtered mail delayed + webmail problems

Posted (October 13th, 2009 at 9:50 am PST) by andrea

If you are using our junk filters to scan your incoming mail, you may be experiencing delays in receiving mail for the past few hours. You also may be having trouble connecting to or staying connected in the webmail interface (regardless of using junk mail or not). Our admins are looking into the problems, sorry about the inconvenience. Please check back here for updates.

UPDATE 11am PST: The mail delay problem has been almost fully resovlved, mail should be coming through as expected. We continue monitoring the servers. As for webmail, it’s still being looked into, sorry about the continued inconvenience.

UPDATE 1:20pm PST: We can now safely say that there are no more delays in receiving your junk-filtered mail. However, the webmail login and connection issue is still apparent, and it is now affecting other clients as well manifesting in multiple login prompts and inaccessible email. We are working on the resolution, thank you for your patience!

UPDATE 4:50pm PST: All email issues have been resolved for a few hours now: no more mail delays and login and sending problems. If you’re still having trouble with your mail, please submit a support ticket with the details, so we can assist you.


10:47 pm

DreamHost Network is down.

Posted (October 12th, 2009 at 10:47 pm PST) by Sandon

It looks like our core router lost its BGP session which took down our entire network (as far as we can tell). We have our network administrator heading to the data center now and should be there in 20 minutes or so. More updates to come.

This is affecting all DreamHost services.

Update 11:20 PM According to our Network Administrator the issue is fixed and all services should be restored. Please send a support ticket if you are still having any issues connecting to any of our services.


6:13 pm

Networking issues causing mysql and web downtime

Posted (October 11th, 2009 at 6:13 pm PST) by andrea

Unfortunately, we’re experiencing some networking issues that are affecting our web (including PS) and mysql servers. If your site is suddenly not loading or can’t connect to the database, it’s most likely related. We’re looking into this and will be posting updates as soon as available. Sorry about the inconvenience!

UPDATE 7:15pm PST: The problem has resolved shortly after this message was posted. We decided to wait a bit longer to make sure all servers are now accessible. Sorry again about the inconvenience this has caused. If you’re still having problems with your domains, please submit a support ticket with the details as it’s likely unrelated. We’ll be happy to assist!


3:16 am

Packetloss on some MySQL’s and one web-server

Posted (October 10th, 2009 at 3:16 am PST) by Sandon

It looks like one of our edge switches is having some problems causing packetloss which is making the machines extremely slow and unresponsive. This is affecting the following machines:

dewy, huey, cupid, marbles, nanook, donald, daffy, tweety, woodstock, psmysql745, psmysql799, psmysql961, psmysql962, psmysql966, psmysql967, psmysql979, psmysql986, psmysql991, psmysql1829, psmysql1861, psmysql3171, psmysql3228, psmysql4588, psmysql4975, psmysql5278, psmysql6455

All of these are MySQL servers except ‘cupid’ which is a webserver. More information and possibly an ETA will come later.

Update 04:36 AM PDT: It looks like the problem was actually with a specific VLAN behind one of our core routers and wasn’t actually the switch itself. One of our senior Network Administrators was able to fix the problem and all services should be restored on the above machines. IF you are still having trouble and on any of these machines please contact support.


3:20 pm

Network problems causing downtime on some homie machines

Posted (October 8th, 2009 at 3:20 pm PST) by andrea

The following machines are affected by the current network outage. Our admins are working on the resolution, please check back for updates and details. We apologize for the inconvenience!

algiers alouettes angels argonauts avengers baku bayhawks bears belmopan bengals bills blackhawks blazers bluebombers bluejays bobcats braves brewers bridgetown broncos browns bruins brumbies brussels bubsy buccaneers bucks buenosaires bulls bullwinkle butterfish canberra cannons cardinals cavs chapulin chivas clippers cobra colts cowboys crew crusaders d-fenders dhaka diamondbacks dodgers dopey dumbo dynamo eagles earthquakes eltigre eskimos expos falcons fever fire flames foghorn force galaxy grinch grizzlies gumby hawks heat herediano highlanders hornets hurricanes indians inki islanders jaguars jazz jets jetsons kabul kings knicks lakers leghorn liberty lightning lions lizards magic magoo manama mariners marlins mets minsk monarchs nassau nationals nets niners nuggets oilers olympiacos outlaws pacers packers padres panthers patriots penguins phillies pirates pistons popeye predators pride pumbaa raiders rams rangers rapids raptors ravens rays reds redwings revolution rockets rockies roughriders royals saprissa seahawks shock slowpoke snoopy sox spaceghost sparks spurs stampeders steelers stjohns suns texans thrashers thunder tigercats tigers tirane titans topcat twins vienna vikings waratahs warriors wizards yankees yerevan zazu and some of our private servers

UPDATE: Shortly after this post, the problems were gone, servers are accessible again. Thanks for your patience! If you’re still having issues, please submit a support ticket with details, as it’s likely unrelated to this network outage.


11:56 am

High loads on some servers

Posted (October 8th, 2009 at 11:56 am PST) by andrea

Our admins are currently looking into the following servers to find out what’s causing the high loads. We apologize for the inconvenience and thank you for your patience while we stabilize the machines:

masseria
Update 5pm PST: this server has been stable for most afternoon, but we’re continuing to look into it

buenosaires
Update 5pm PST: we’re continuing to look into what’s causing the high load on this server

luanda
Update 5pm PST: this has been stable all afternoon, but we continue keeping an eye on it to make sure whatever caused the load doesn’t return

tirane
Update 5pm PST: we are in the process of offloading half the accounts on this server to another machine

manama
Update 5pm PST: this server has been stable for most afternoon, but we’re continuing to look into it


7:28 pm

DreamHost Central Database Maintenance, Thursday 10pm PDT

Posted (October 7th, 2009 at 7:28 pm PST) by DreamHost

We will be performing some maintenance on our central database servers tomorrow night, Thursday Oct 8, starting at 10PM PDT, ending at 11:59PM PDT. This will take the following services offline:

DreamHost Web Panel
Signups
Domain Registrations
DreamHost-provided Webmail
DreamHost Wiki

We’re sorry for any inconvenience this causes.


6:22 pm

Bulger web server is going offline for file system check

Posted (October 7th, 2009 at 6:22 pm PST) by andrea

Unfortunately, we have to bring down the server to run a file system check as it seems to have developed some problems. This is a very large machine, and it may take a while to go through it. We apologize for the inconvenience this is causing. Please check back for updates.

UPDATE 6:50PM PST: The machine is back from the file system check and it seems to have resolved the problems. Thanks for your patience during this process!


2:15 pm

Web server ‘luanda’ is moving to new hardware

Posted (October 7th, 2009 at 2:15 pm PST) by andrea

We’ve been closely monitoring luanda since we started moving accounts to it a few days ago. It’s been having load issues that we’re looking into (part of it is due to the ongoing moves that require syncing data and hence take up lots of bandwidth at a time). We’ve have found some culprits, but we’re unable to closely examine them as the machine is now having hardware issues and shuts itself off. We’re moving it all to new hardware and will continue looking into the unusually high load.

We apologize for the inconvenience this is causing, and please check back here for updates.

UPDATE 2:25pm PST: The machine is on new hardware now, and it’s loading up fine. We continue the monitoring, and won’t resolve this post until it’s completely stable again. Thanks for your patience.

UPDATE 3:50pm PST: The machine has been stable since being moved to new hardware, but we’ll continue monitoring it throughout the day.

UPDATE 6:00pm PST: We’re resolving this post as the machine has been constantly stable since we moved it to new hardware. If you experience any problems with your hosting, please don’t hesitate to let us know by submitting a support ticket with the details.


 
© 1996-2009, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
10:21 am

Mail originating from webserver is being delayed

Posted (October 6th, 2009 at 10:21 am PST) by brians

Mail is being queued on the relay machine that handles outgoing messages that are sent from your mailserver. As a result, any message that’s generated by your website ( ex. online form that uses php mail() command ) will be delayed. Our administration team is diligently working toward resolving this issue. We sincerely apologize for the inconvenience this has caused and appreciate your patience. Please check back for future updates.

UPDATE 10:55 AM PST: Our administration team has identified the source of the problem. There was a spam run on one of the webservers. We are in the process of deleting all of the spam messages. Removing these messages could take some time to complete. Again, your patience is truly appreciated.

UPDATE 12:40 PM PST: The spam messages have been cleared out and IPs involved restricted by our admins. Mail originating from our web servers should be going out again without delay. If you notice any more issues, it’s most likely unrelated, so send our support team a ticket with details.


11:59 pm

Web Panel Downtime

Posted (October 5th, 2009 at 11:59 pm PST) by patrickm

We apologize, as we are experiencing an issue with our central database server, which is causing downtime with the customer Web Panel.  We are working to resolve this issue quickly and hope to have everything back up shortly.  We will update this post when the problem is resolved.

UPDATE: This issue is now resolved.  Please contact support if you continue to have any issues!


4:00 pm

Downtime issues in the postal cluster

Posted (October 5th, 2009 at 4:00 pm PST) by andrea

We’re experiencing some networking-related downtime on some of our postal cluster web servers. Our admins are looking into it, and we’ll post an update as soon as possible. We’ll try to post an updated list of the affected machines soon. Please check back here often.

UPDATE: The currently affected machines are some of the postal mail servers and gloop annie glass monkey dib gotcha lenny hibbert krusty mcclure marge jubjub millhouse larry ralphie cheeky lisa skinner wiggum burns kearny cletus apu willie seamus.

We apologize for the inconvenience.

UPDATE 5:35pm PST: We have restarted the affected machines and they are all coming back one by one. We’re keeping an eye on them and won’t resolve this post until they are all back up and operational. Thank you for your patience.

UPDATE 6:23pm PST: The servers all came back fine from the restart, but we’re still dealing with random networking issues whcih are affecting access and speed. Sorry about the continued inconvenience, we’re working as hard as we can to get this resolved for you.

UPDATE 9pm PST: As of about two hours ago we consider the situation resolved. The servers are back and stable. If you’re still having problems with your domains, they are likely unrelated, so please open a ticket with our support team to get it looked at. Thanks for your patience!


12:48 pm

Tirane server issues

Posted (October 5th, 2009 at 12:48 pm PST) by danwy

The tirane server has not been having a good morning. Its load has been spiking and we have yet to find the cause. Unfortunately, we don’t have much information about it right now. This post is just to let you know that we are aware of the issues and are working on them as soon as we possibly can. We’ll update this status post as soon as we have any additional info. We’re sorry for any inconvenience this causes you.

Update: 4:20pm (PST) – Things are looking better on the server. We’ve evaluated the hardware and system software that we’re running on the server, as well as the resource consumption on it and everything looked fine. At this point we believe that it may be a customers site that is tying up resources and causing the load to spike. We were able to narrow things down to some specific services, but not specific domains yet. When we stopped the services to investigate further the load dropped and hasn’t risen since. If the load does creep back up we do have a few things that we can check on which we believe are responsible for the issues on this server. We apologize for the inconvenience this has caused you. We don’t like your sites being down any more than you do, but this has been an elusive issue that has been difficult to pinpoint.

Update: Oct 7 4:00pm (PST) – We’re sorry for the lack of updates regarding this issue, but there hadn’t been any new developments over the last day or two until now. We believe that we may have found several users on the machine contributing to the issue for the last couple of days. This was made especially tricky because once we believe we found the issue another one would pop up in its place to start causing issues. We’ve disabled a handful of users that we believe were contributing to the problems and the server load is at a much better load now. We’re going to be re-enabling those users one at a time over an extended period of time in the next 24 hours in hopes of pinpointing the exact cause. At this point things seem to be stable on the server, but we’re going to continue to work on it until we’re sure that it will remain that way. We understand the frustration this has caused you and was equally as frustrating (if not more so) for us to not be able to find the problem in a timely manner. We’re going to keep a close eye on this server until we’re certain that the issue has been completely addressed.


8:59 pm

New domain registrations and new transfers temporarily unavailable

Posted (October 3rd, 2009 at 8:59 pm PST) by andrea

We’re experiencing problems with our domain registration lookups, so currently it’s not possible to register a new domain or initiate a new registration transfer through our web panel. We apologize for the inconvenience. Please check back for updates.

Update 9:30pm PST: Looks like we’re having trouble with our backend and Verisign. Unfortunately, only Verisign can correct this issue but they aren’t able to until the morning. It looks like new registrations and transfers (renewals as well) will be down until then but check back for updates!

Update 12:15pm PST: Unfortunately, we’re still waiting to hear back from the registry regarding this. Until we hear from them, there’s not a whole lot we can do regarding this because the solution needs to come from their end. As soon as they contact us we should have this issue resolved.

Update 8:55pm PST: We have finally gotten the issue resolved with Verisign and domain registrations/transfers/updates etc are now in working order! We apologize for the inconvenience and if you’re still experiencing issues, please contact our support staff immediately!


5:05 pm

Quick Power Strip Swap

Posted (October 2nd, 2009 at 5:05 pm PST) by Leon

Tonight at about 9:00pm PST, we will be swapping out a power strip in one of our racks. This should only cause about 10 minutes of downtime but we’re making the downtime window about 1 hour just in case! The following servers will be affected by this:

rays
pirates
rangers
cardinals
nationals
nets

Update 9:10PM PST: The swap was successful and all of the servers mentioned above came back up without incident. Services on the machines are still starting but for the most part, all of your websites should be back up in the next couple of minutes. The total downtime was around the 10 minute estimated downtime period. If you’re still experiencing issues, please contact our support team!


8:11 am

unexpected downtime with a few Spunky domains

Posted (October 2nd, 2009 at 8:11 am PST) by patrickm

In an effort to shore up all the additional IPs in our system, we reclaimed a bunch of old IPs that were no longer in use.  Unfortunately, a few Apache services did not receive a proper update and the old IP was left active.  We are working our way through these services now and hope to have them all fixed up quickly.

Unfortunately, it’s not possible to provide a list of the affected domains, but we don’t expect this issue to last much longer, as we have multiple techs addressing it at this moment.  We will update this post once this process is complete.  If you are experiencing downtime with your domain, please feel free to contact Support for additional info!

UPDATE: This issue is limited to the Spunky cluster of servers, actually.  You can tell if you’re on Spunky by using the “Account Status” drop-down in your Web Panel and checking the “Email Server” item.


3:22 pm

Rothstein and Mangano going down momentarily!

Posted (October 1st, 2009 at 3:22 pm PST) by Leon

I will be replacing the RAM in just a moment and they will be down for about 5-10 minutes apiece if everything goes smoothly.

Edit 4:02: Done deal! If you have any problems, please contact your friendly DH technical support person.


1:36 am

Cruisin is Cruisin’ no more.

Posted (October 1st, 2009 at 1:36 am PST) by Sandon

The web-server ‘Cruisin’ was having some weird issues and would not reboot. After power-cycling it the machine did not come back up and after taking one of our NOC personnel checked it out he came to the conclusion that the hardware is bad so we are getting it moved to a new set of hardware. This process typically takes 30 minutes to 1 hour (sometimes longer in some situations). This is affecting you if your webserver is listed as ‘Cruisin’ From the “Account Status” link from the panel.

More updates to come.

Update 02:01 AM PDT OK the server is all back up and serving files again. If you are still having any issues and your sites are hosted on ‘cruisin’ please contact our support team.


10:00 pm

Quick postal file server downtime 2009-09-30 at 10pm!

Posted (September 30th, 2009 at 10:00 pm PST) by kitchen

In order to finish clearing out a rack so it can be repurposed, we need to move the remaining file server it contains to another rack. This will involve shutting down the system while it is moved, during which time the data on the server will not be available. We anticipate this will take approximately 30 minutes and we will update here when everything is done.

We will be performing this maintenance at 10pm pacific time on 2009-09-30.

Thank you for your patience and we apologize in advance for any inconvenience this may cause.

Update: 10:03pm: we are starting this maintenance now, see you on the flipside!

Update: 10:17pm: the server is back up and running and things appear to be fine. We’re going to run some more checks before we call it totally good!


 
© 1996-2009, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
2:52 pm

RubyGems updated to 1.3.5 for all servers now!

Posted (September 30th, 2009 at 2:52 pm PST) by DreamHost

If you haven’t already noticed, RubyGems 1.3.5 should now be deployed to all servers at this point! If your server isn’t running 1.3.5, please open a support ticket and we’ll investigate. As mentioned in the previous post about this when we did a partial deployment, the upgrade from 1.3.1 shouldn’t cause any major problems for applications using gems. If you had previously been trying to use Rails 2.3.4 and were experiencing errors saying RubyGems 1.3.2 or higher was required, that should now work fine!


8:02 am

Garland Hardware Failure

Posted (September 29th, 2009 at 8:02 am PST) by Kyle

The hardware running the “garland” webserver is being stubborn as of late so we are migrating it to a different physical machine this morning.  Estimated downtime is 30 minutes.

Update 8:58AM PST: The configuration updates finished about 20 minutes ago, if you are still having an issue with one of your sites please contact our support team!


5:55 am

Fitzpatick having writing problems.

Posted (September 29th, 2009 at 5:55 am PST) by Javier

Unfortunately our server Fitzpatick is having some issues at the moment which has caused it to need some hardware replacement, this has also caused a problem were users are not able to write to their directories. We are having our admins work on this and will update this post once there is more information.

Update: 09-29-2009 – Our admin has been able to correct the writing issues that the server had. If you are still having problems please contact support


12:30 am

jubilee exploded

Posted (September 29th, 2009 at 12:30 am PST) by Sandon

The web-server ‘jubilee’ had a NIC failure so we are moving her to new hardware. Downtime will probably be around 1 hour once the move is complete. More updates to come.

Update 01:47 AM PDT Jubilee is back up and running as expected. If you are still having issues and are on jubilee please contact support.


6:09 pm

Shared Database Outage

Posted (September 28th, 2009 at 6:09 pm PST) by Patrick

We recently experienced a minor MySQL outage affecting a very small subset of customers on the ‘homie’ cluster. This outage was a direct result of an oversight on my part. We are renumbering some services to reclaim wasted subnets. During this process, a fat finger resulted in a few services not getting restarted after their IP addresses were changed.

All affected databases have been back up and functioning properly for an hour or so. If you are still experiencing please contact support.

–UPDATE Mon Sep 28 18:25:17 PDT 2009–

To avoid confusion, the affected machines (not all services on these machines, just a handful):

huey
lady
woodstock
balto
cabbit
schnoodle
imu
shamu
wolpertinger
dewey
marbles
nanook
peabody
tweety


8:18 pm

Panel Madness

Posted (September 26th, 2009 at 8:18 pm PST) by JeffH

Nothing like March Madness, unfortunately.   Our web panel is down at the moment (as I’m sure anyone looking at this post already knows), and it’s causing no small amount of headache for our developers and admins!  We are working to get everything back up and running happily in an expedient manner, but please bear with us while we sort out the messy bits.  Will update this post as soon as the panel is back up, that we may tell you what you already know!  We apologize for any inconvenience!

Update!  Posted (September 26th, 2009 at 8:19 pm PST) by JeffH

Panel is still down, and we are still working on it!  This affects: Everyone  using the panel.

Update 9:08PM PST: The panel should be back in action! There were some issues with one of the database servers which seems to be resolved now.

-ants


3:18 pm

Swampthing swamped!

Posted (September 26th, 2009 at 3:18 pm PST) by mir

It ran out of memory causing a load of over 5000 – unfortunately it was not responsive so we had to power cycle it and it is coming back up now. It appears that is is checking disks so it is not popping right up – we apologize for the inconvenience and will update this post as soon as it’s back online.

Update 4:33PM PST: The file system check came back clean and the server is now back online. We apologize for the inconvenience!


7:02 pm

RubyGems updated to 1.3.5

Posted (September 23rd, 2009 at 7:02 pm PST) by DreamHost

We just upgraded RubyGems to version 1.3.5 on many of our shared hosting servers. If you were previously experiencing problems getting your Rails 2.3.4 application to work due to a RubyGems compatibility issue, it should work now! The affected machines are as follows:

alouettes angel angels apocalypse aquaman argonauts avalanche avengers banshee batman bayhawks bears bengals bills bishop blackhawks blade blazers bluebombers bluejays bobcats bonanno brasco braves brewers broncos browns bruins brumbies bruno buccaneers bucks bugsy bulger bulls bullseye cable cannonball cannons capone cardinals castellano catwoman cavs cerebus chivas clippers cloak colossus colts constantine coolblue coonan cowboys crew crusaders cummiskey cyclops dagger dalitz deadpool deathlok diamondbacks dillinger dodgers dredd dynamo eagles earthquakes eastman eskimos exodus expos falcons fay fever fire fitzpatrick flames floyd force forge galactus galaxy gambit genovese ghostrider giancana greengoblin greenlantern grizzlies haddock havok hawkeye hawkman hawks heat herediano highlanders hobgoblin hornets hulk humantorch huntress hurricanes iceman indians islanders jaguars jazz jets johnnyblaze joker jubilee kakuji kingpin kings knicks lakers larchmont legs liberty lightning lions lizards luthor magic magneto mariners marlins marv masseria mclean mekon mets monarchs mothman mrfantastic mystique namor nationals nets nightwing niners nitti nuggets obelix oilers olympiacos outlaws pacers packers padres panthers patriots penguin penguins phantom phillies pirates pistone pistons preacher predators pride professorx proty quicksilver racers raiders rams rangers rapids raptors ravens rays reds redwings revolution riddler robin rockets rockies rogue roughriders royals sabretooth sandman saprissa seahawks shadow shock silversurfer snocap snowbird sox sparks spawn spectre spiderman spillane spurs stampeders starfire steelers storm sugeknight suns sunspot supergirl swampthing terranova texans thing thor thrashers thunder tick tigercats tigers titans trafficante twins twoface venom vikings walsh waratahs warmachine warriors wasp wilshire wizards wolfsbane wolverine wonderwoman yankees zagnut zatara zwerbach

If you run into any problems, please open a support request via your WebPanel.


5:05 pm

Glass Shattered

Posted (September 23rd, 2009 at 5:05 pm PST) by Leon

It seems like the glass web server blew its boot drive which is causing some issues with services on it. We’re currently working on moving it to new hardware and hope to have all services on it back up in about an hour. Please check back for updates!

Update 6:43PM PST: Sorry that it took longer than expected. In any case, the server should be back in action! If you’re still experiencing issues with the glass server, please contact support.


4:24 pm

Passenger upgraded to 2.2.4 on older servers

Posted (September 23rd, 2009 at 4:24 pm PST) by DreamHost

Over the last several weeks we’ve been slowly updating Passenger on our older servers from version 2.0.6 to 2.2.4. Yesterday we finished updating the remaining machines running Passenger 2.0.6. So, all our servers should now be running with Passenger 2.2.4 or higher.

There are a couple things this might affect if you were using Passenger. If your Rails site suddenly stopped working make sure you check your public directory and rename or remove any “.htaccess” file that’s there. Passenger 2.0.6 ignored .htaccess files completely, but 2.2.4 reads that file and the default .htaccess file Rails used to generate causes things to break under the newer version of Passenger. If that doesn’t resolve your problem, please open up a support request via your WebPanel.


 
© 1996-2009, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
1:52 am

PostalMail authentication issues

Posted (September 23rd, 2009 at 1:52 am PST) by Sandon

It looks like our central MySQL server which handles authentication for our ‘postal’ mail cluster also listed as ‘sack’ as “your mail server” from the account status in the panel is having some issues causing mail authentication to fail for sending/receiving mail. An admin is looking into the issue and we hope to get it back up ASAP.

1:55 AM PDT: Ok all should be well again. If you are still having any issues authenticating for mail and you are in the postal mail cluster (your mail server listed as sack) please contact support.

2:12 AM PDT: It looks like the postal cluster is now fixed but now our homie-cluster is having some issues which might be related to the fix. This would be affecting you if your mail server has ‘homie’ in its name.

2:18 AM PDT: Ok, everything should be fixed now.


3:19 pm

bruins on the sideline

Posted (September 21st, 2009 at 3:19 pm PST) by danwy

The server ‘bruins’ has been having some issues today that we’ve had a hard time pinning down. This is resulting in an elevated load and is causing sites that are hosted on it to either be slow or unavailable. Our admin team has been investigating the cause, but haven’t been able to pinpoint it yet. We are doing all that we can to diagnose the problem on this server and get it running smoothly again. We will update this post once we feel that it is stable again. We’re sorry for any inconvenience this causes you.

Update 9/22 – 11:15am (PST): Unfortunately, the server is still having some intermittent issues. We thought we had it nailed down yesterday afternoon, but the load has again spiked as of this morning. The issues are definitely puzzling, as all of its resources are well within acceptable levels. There’s still plenty of memory, only half of the cpu is being used, network cards are performing optimally and there are no resource heavy scripts running that we can find. I’m sorry that we don’t have better news at this time, but our admin team is still trying to get to the bottom of these issues.

Update 9/24 – 2:40pm (PST): Sorry for the delay in updating this. We replaced the kernel on this machine yesterday and it seems to be running a lot better since then. At this point, the issues do seem to be resolved, but please contact support if you have any further issues on this server.


2:43 pm

Spacey mail cluster errors.

Posted (September 18th, 2009 at 2:43 pm PST) by mir

One of the mail servers in the spacey cluster is experiencing issues – its IMAP client having problems binding to the port it uses which is resulting in connection/login issues for customers. The admin team is fixing this right now and should have it corrected shortly (15-30 minutes). We’ll update this post as soon as we have more information.

Update: 15:26 This problem is resolved now. The machine we were having problems with was actually a machine replacement. The replacement machine was unfortunately missing some packages that were needed on our mail machines. The problem was intermittent because we load balance our email. Sorry about this. Everything should be resolved now. We have tested accounts that have written in and so far everything looks great.


10:06 am

Help me Kenobi you are my only hope!

Posted (September 18th, 2009 at 10:06 am PST) by Kyle

One of the partitions on kenobi is suffering from corruption so we are moving it over to newly imaged hardware. Approx. downtime should be 30-40 minutes.

Update 11:10AM PST: as of about 20 minutes ago the server has been running from it’s new home and everything is looking good so far.  If you are still having trouble with one of your domains please contact support!


7:01 am

Manjula needs to man up.

Posted (September 17th, 2009 at 7:01 am PST) by Kyle

The hardware running the shared server “manjula” went dark, after a quick power supply swap it still will not power on so we are in the process of migrating services to another machine.  Estimated downtime is 45 minutes to 1 hour, sorry folks!

Update 8:25AM PST: The machine is still getting configuration updates, some services are already restored and others will be up shortly.  We will provide another update when everything is 100% complete, sorry for the delay!

Update 9:00AM PST: The takeover reconfigurations finished at approx 9AM, everything should be restored to normal so if your still having trouble with one of your domains please contact support!


5:21 am

panel outage

Posted (September 15th, 2009 at 5:21 am PST) by patrickm

We are very sorry to report that an internal issue is causing the customer Web Panel and WebMail to be unresponsive at this time.  We are working to restore service and will update this post once we have more information!  We sincerely apologize for this outage and hope to have everything back up soon.

UPDATE 9/15 6:16AM (Pacific): This issue is now resolved.  We are still investigating the cause and hope to be able to provide more details soon.  If you are still unable to access the Web Panel or your WebMail accounts, please contact Support and we’ll be happy to help!


11:36 am

homie-vserver76 having problems

Posted (September 14th, 2009 at 11:36 am PST) by danwy

One of our PS host machines is having some issues and is currently being investigated by our admins. Here is a list of servers affected:

ps13053 ps13054 ps13055 ps13057 ps13058 ps13097 ps13099 ps13101 ps13104 ps13107 ps13108 ps13109 ps13110 ps13111 ps13112 ps13113 ps13114 ps13115 ps13116 ps13117 ps13119 ps13120 ps13121 ps13122 ps13125 ps13126 ps13127 ps13128 ps13129 ps13130 ps13133 ps13201 ps13209 ps13531 ps13537 ps13552 ps13561 ps13601 ps13631 ps13641 ps13650 ps13706 ps13707 ps13708 ps13710 ps13711 ps13716 ps13717 ps13719 ps13739 ps13753 ps13764 ps13778 ps13786 ps13855 ps13887 ps13888 ps13900 ps13912 ps13921 ps13926 ps13935 ps14003 ps14024 ps14084 ps14140 ps14150 ps14160 ps14164 ps14169 ps14170 ps14171 ps14177 ps14178 ps14191 ps14235 ps14268 ps14338 ps14347 ps14351 ps14366 ps14367 ps14370 ps14378 ps14379 ps14384 ps14423

If you’re on one of these servers and your site is not working or you can’t access your server, we are working on getting it back up as soon as we possibly can. We’ll post an update here once we have more information. We’re sorry for any inconvenience this causes you.

Update – 3pm (PST): This issue should have been resolved about an hour ago. If you’re on one of the PS servers mentioned above and are still having issues, please contact support so that we can investigate it right away.


5:53 pm

homie-vserver90 getting new hardware

Posted (September 12th, 2009 at 5:53 pm PST) by Leon

Recently, homie-vserver90 has been looking like it’s been having raid issues, so we’re moving it to new hardware. We don’t expect much downtime but the window is about 1 hour starting now. If your machine is one of the following, you will be affected:

ps14427 ps14433 ps14498 ps14509 ps14515 ps14521 ps14528 ps14534 ps14541 ps14547 ps14553 ps14558 ps14563 ps14567 ps14571 ps14577 ps14581 ps14585 ps14591 ps14602 ps14609 ps14616 ps14623 ps14634 ps14640 ps14646 ps14651 ps14656 ps14846 ps14848 ps14851 ps14854 ps14857 ps14860 ps14864

Update 7:47PM PST: Sorry that it took longer than expected but we hit a configuration snag on boot when moving the drives over to its new hardware. In any case, your Private Servers should be back up but if you’re experiencing any issues and your server is listed above, please contact support!


3:32 pm

Server Moves for Spunky Cluster!

Posted (September 10th, 2009 at 3:32 pm PST) by micah

We are currently working to move customers from our no-longer-scaling NFS based hosting servers, to our local storage servers. This move is all done behind the scenes by copying (and recopying) and checking (and rechecking) each customer’s data and then updating the users and domains in our databases.

There should be very little if any fallout from this process unless you happen to be hosting your DNS elsewhere. If you do have external DNS for any of your domains, you can contact support to find out your new IP so you can update your records OR you can lookup your DNS via our control panel yourself by going to menu option (DOMAINS > MANAGE DOMAINS) and clicking on the “DNS” link for the domain in question. You should receive an email when the move is complete. To reiterate, this affects customers in the spunky cluster only, other clusters will be migrated later.

Also, this move entails an upgrade from Debian 3.1 (sarge) to Debian 4.0 (etch) which means quite a few pieces of software your site may rely on will be upgraded as well. Any custom compiled software will likely need to be recompiled as well due to the change in OS (and architecture). If you encounter any problems or have any questions please contact our support team!


3:58 pm

patriots under the weather

Posted (September 9th, 2009 at 3:58 pm PST) by danwy

Many of you on the patriots server may have noticed some issues with it today. It looks like the the servers disk has filled up and the data moves that we started earlier in the day to free up some space have failed. With the disk being full, it is causing the load to shoot up and make anything running on it either slow or unresponsive. Our admin team is currently in the process of manually moving off some of the larger user accounts and will be keeping a close eye on it to make sure it completes. Unfortunately, we don’t have an ETA on when this will be complete, but our admins are doing all that they can to restore stability to this server. We’ll update this status post as soon as we have any additional information. We apologize for any inconvenience this causes you.

Update 4:29pm (PST): Things are looking better on this server now. We were able to free up some space while we wait for some of the data migrations to finish, which should resolve the issue permanently. If you have any further issues with this server, please contact support to let us know.


 
© 1996-2009, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
5:29 pm

DoS attack causing connection issues

Posted (September 7th, 2009 at 5:29 pm PST) by mir

for customer on one of our PS hosts, the effected guests are:

ps10064 ps10066 ps10105 ps10108 ps10119 ps10300 ps10305 ps10307 ps10377 ps10602 ps10609 ps11352 ps11353 ps11650 ps12001 ps8461 ps8462 ps8463 ps8464 ps8465 ps8466 ps8467 ps8468 ps8469 ps8470 ps8471 ps8472 ps8473 ps8474 ps8475 ps8476 ps8477 ps8478 ps8479 ps8480 ps8481 ps8482 ps8483 ps8484 ps8485 ps8486 ps8487 ps8488 ps8489 ps8490 ps8491 ps8492 ps8493 ps8494 ps8495 ps8496 ps8497 ps8498 ps8499 ps8500 ps8501 ps8502 ps8503 ps8504 ps8505 ps8506 ps8507 ps8508 ps8509 ps8510 ps8511 ps8512 ps8513 ps8514 ps8515 ps8516 ps8517 ps8518 ps8519 ps8520 ps8521 ps8522 ps8523 ps8524 ps8525 ps8526 ps8527 ps8528 ps8529 ps8530 ps8531 ps8532 ps8533 ps8534 ps8535 ps8536 ps8537 ps8538 ps8539 ps8540 ps8541 ps8542 ps8543 ps8544 ps8545 ps8546 ps8547 ps8548 ps8549 ps8550 ps8551 ps8552 ps8553 ps8554 ps8555 ps8556 ps8557 ps8558 ps8559 ps8560 ps8561 ps8562 ps8563 ps8564 ps8565 ps8566 ps8567 ps8568 ps8569 ps8570 ps8571 ps8572 ps8573 ps8574 ps8575 ps8576 ps8577 ps8578 ps8579 ps8581 ps8582 ps8583 ps8585 ps8587 ps8588 ps8589 ps8591 ps8592 ps8593 ps8594 ps8596 ps8597 ps8598 ps8600 ps8601 ps8602 ps8603 ps8604 ps8606 ps8607 ps8608 ps8609 ps8610 ps8611 ps8612 ps8613 ps8614 ps8615 ps8616 ps8617 ps8618 ps8619 ps8620 ps8621 ps8622 ps8623 ps8624 ps8625 ps8626 ps8627 ps8628 ps8629 ps8630 ps8631 ps8632 ps8633 ps8634 ps8635 ps8636 ps8637 ps8638 ps8639 ps8640 ps8641 ps8642 ps8643 ps8644 ps8645 ps8646 ps8647 ps8648 ps8649 ps8651 ps8652 ps8653 ps8658 ps8967 ps8968 ps8971 ps8972 ps8973 ps8975 ps8976 ps8979 ps8981 ps8983 ps8985 ps8987 ps8991 ps8993 ps8994 ps8996 ps8998 ps9009 ps9010 ps9123 ps9124 ps9125 ps9126 ps9127 ps9128 ps9129 ps9130 ps9131 ps9133 ps9136 ps9142 ps9146 ps9151 ps9155 ps9342 ps9540 ps9542 ps9574 ps9576 ps9579 ps9584 ps9589 ps9608 ps9805 ps9807 ps9831 ps9833 ps9837 ps9842 ps9853 ps9876 ps9893

We have identified the target and are in the process of addressing the issue. We’ll update this post once it is resolved.

UPDATE: This incident was resolved shortly after our initial post. Our administration team will continue to actively monitor. Again, we sincerely apologize for the inconvenience this has caused and appreciate your continued patience.


1:36 am

Jubilee Lost its Spark

Posted (September 5th, 2009 at 1:36 am PST) by Leon

The jubilee webserver is currently having issues with its hardware and we’re working on bring it back up. Unfortunately, we do not have an ETA as to when it will be up but we hope to have it up soon. Please check back for updates!

Update 3:08AM PST: Unfortunately, our efforts to bring the server back on its original hardware and swapping its hard drives over to another machine have failed as the raid array is hosed. We’re currently getting a new server up and will be restoring user data from backups. We apologize for the inconvenience but please check back for updates.

Update 5:16AM PST: The machine is back up and user data mounted from backups. All final configuration files and other things are being pushed to the server and all sites should start comming back up within the next 30-40 minutes (some are already back up).

Update 6:05AM PST: Things are going a bit slower than expected due to the slower mounts to our backup server but more than half of the sites should be back up now.

Update 11:45AM PST: Things are still chugging along and configs on the server are still running but your sites should be up soon with the restore going on. Again, we apologize for the inconvenience this is causing you.

Update 2:00PM PST: Configs have finished running and we have mounted user data from the backups so that your sites are functional while the data gets copied over locally to the new server. For those of you wondering, the backups for the machine are about 24hours old give or take.

Update 9:53pm PST: The restore is still chugging along but again, your websites should be functional during this time! You may notice intermittent downtime every so often while the restore runs but for the most part everything should be online for you. Check back for updates!

Update 09/06/2009 9:10PM: We’re having a minor hiccup with the backup machine and we’re working on correcting it. Should hopefully have everything back up and running soon!

Update 09/06/2009 9:55PM: Backup machine is now functioning correctly and the restore has resumed. Again, your websites will be functioning normally while the backups are being restored as they are mounted on the machine as well.


8:14 am

SMTP (Mail sending) issues on Spunky and Homie clusters.

Posted (September 4th, 2009 at 8:14 am PST) by mir

We are aware of the issues that customers are experiencing when trying to send mail from these clusters and our admin team is investigating the matter. We’ll update this post as soon as we have detailed information to report.

UPDATE 09:48 PM PST: Our administration team have found the source of the problem. There were faulty network addresses in the Postfix configuration file. The network addresses has been removed and Postfix has been restarted. We will continue to actively monitor the service. Again, we truly apologize for the inconvenience this has caused and appreciate your continued patience. Of course if you continue to experience this behavior, please do not hesitate submitting a support request.


6:40 am

Network cutover causes some connection issues

Posted (September 4th, 2009 at 6:40 am PST) by mir

At around 12:30 AM PST this morning there was a hiccup with some networking changes which caused some customers to experience connection issues. This was resolved and we are currently clearing out the related support issues. Due to the nature of the project this is not something that will be a recurring problem.


12:55 pm

Delayed mail originating from sites

Posted (September 3rd, 2009 at 12:55 pm PST) by danwy

One of our mail servers that relays mail originating from websites hosted on our system is having some issues with its mail queue. As a result, any mail being sent from web servers is being delayed. We are aware of this situation and one of our mail admins is currently working on resolving this issue. We don’t have an ETA when this will be resolved yet, but we will update this post with any additional information as soon as we have it.


11:58 am

Hosting server Fud having issues

Posted (September 3rd, 2009 at 11:58 am PST) by mir

Due to a bad NIC (Network Interface Card). We’re deploying our network operations team to fail this over to new hardware immediately and should have it resolved in the next hour. We’ll update this post once it’s taken care of.

UPDATE 1:27PM PST: Fud is now migrated onto new hardware and services have been restored. If you are still experiencing problems with a domain serviced by Fud please contact support!


10:25 am

Avalanche back online.

Posted (September 2nd, 2009 at 10:25 am PST) by mir

We had to take this machine offline while we investigated the source of widespread website defacement. Fortunately we were able to determine that the machine was not completely compromised and are now in the process of cleaning up the mess that was made. We apologize for not providing information sooner but due to the sensitive nature of the situation we wanted to complete our investigation before disclosing details.

Update: the vulnerability has been confirmed as fixed (we had actually previously addressed it but this specific machine had become vulnerable after a reboot – the cause of that has been corrected and will not be a cause for concern any longer). We are now in the process of cleaning up and our Abuse team will be contacting customers who were effected.

Update: due to the widespread effect of the attack we are in the process of developing a global fix that we can apply rather than fixing effected users individually


7:47 am

Private Server downtime.

Posted (September 2nd, 2009 at 7:47 am PST) by mir

We are working on restoring one of the Private Server hosts currently – if your PS (web, not MySQL) is unavailable and unresponsive to your reboot request it’s possible that it is related to this matter. We’ll update this post as soon as we have an ETA on the fix and will be linking customers to this status post when appropriate.

Update: This should be resolved in the next 10 minutes or so (fortunately our admin team was able to get it back up and running without having to move it to new hardware which greatly improved the ETA). The guests are coming up now so it is resolved (just not completely back up yet). We’ll be keeping an eye on it to be certain but it looks good!


10:33 am

Network upgrades Sept 3

Posted (September 1st, 2009 at 10:33 am PST) by DreamHost

This Thursday, September 3rd and 8:30 PM pacific time we will be performing a network migration to some new routing hardware, as well as a operating system upgrade on one of our important switches to hopefully address a bug in the software. This will in theory not cause much if any customer facing downtime, but we wanted to warn you just in case! This window runs until Midnight September 4th pacific time.

If anything does go drastically wrong, all services will be affected, but no data should be lost.

Update September 3rd 10:58 PM PDT: It looks like things aren’t going quite as smoothly as expected and we are seing some packetloss and intermittent connectivity to all of our network. We are hoping it should get better or be mostly resolved in the next 20 minutes or so.

Update 11:24 PM PDT: Ok it looks like the network issues have been fixed. We still have some updates to do but they should be transparent and not cause any more packetloss/network issues.

Update 12:10 AM PDT: Ok, all updates have completed and things are back to normal as of about 20 minutes ago.


1:40 pm

Popcorn Popped Its Kernel

Posted (August 29th, 2009 at 1:40 pm PST) by Leon

The popcorn webserver is having issues with its hardware and we’re working on moving it to new hardware. We estimate everything to be back up within the next hour or so but please check back for updates!

Update 3:18pm PST: Sorry that it took longer than expected but popcorn is back up! If you’re still experiencing issues with the popcorn server, please contact support!


 
© 1996-2009, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
11:30 am

Private MySQL Vserver Downtime

Posted (August 28th, 2009 at 11:30 am PST) by Terri

One of our Private MySQL servers is down, and right now we’re troubleshooting the problem. Here is a list of the affected private servers:

psmysql1014 psmysql1479 psmysql1801 psmysql1832 psmysql1872 psmysql1886 psmysql1887 psmysql2430 psmysql2431 psmysql2433 psmysql2435 psmysql2674 psmysql2676 psmysql2677 psmysql2846 psmysql298 psmysql3196 psmysql3398 psmysql3733 psmysql3812 psmysql383 psmysql387 psmysql389 psmysql3899 psmysql391 psmysql3933 psmysql3941 psmysql401 psmysql4177 psmysql4244 psmysql4600 psmysql4688 psmysql4689 psmysql4736 psmysql4873 psmysql5726 psmysql6729 psmysql6760 psmysql6762 psmysql6824 psmysql6860 psmysql746 psmysql800

These private servers will be completely down.

We apologise for the downtime. We will update this space as soon as possible.

UPDATE 12:20 p.m. PST:
The machine’s hardware is not coming back up. Right now we’re copying data to a new machine. This will take some time, but when it is done performance should be better than ever before.

Update 2:33PM PST: We have managed to move the drives over to a new machine which seems to have worked and are working on getting your Private Servers back online. It’s going to be about 30 minutes before everything is back up on the server but once they are, we’ll post an update!

Update 2:54PM PST: All MySQL Private Servers listed above should now be up and running! If you’re still experiencing issues and your MySQL PS was mentioned, please contact support!

-Ants


9:41 pm

Twoface Flipped Its Coin

Posted (August 27th, 2009 at 9:41 pm PST) by Leon

The web server twoface is currently experiencing hardware issues and we’re working on getting it back up as soon as possible. Check back for updates!


8:45 pm

Power Strip Maintenance

Posted (August 21st, 2009 at 8:45 pm PST) by Leon

Due to an issue with a power strip in one of our racks, we will be replacing it tomorrow night 08/22/09 at roughly 7:30PM Pacific time. This will only affect users on private servers and we expect the downtime to be about 10 minutes while the strip is replaced and the servers are plugged back in. The following PS guests will be affected:

ps12642 ps12650 ps12659 ps12665 ps12675 ps12694 ps12700 ps12707 ps12710 ps12726 ps12729 ps12732 ps12736 ps12740 ps12744 ps12749 ps12754 ps12778 ps12782 ps12785 ps12791 ps12793 ps12828 ps12846 ps12852 ps12855 ps12868 ps12870 ps12871 ps12874 ps12876 ps12877 ps12881 ps12890 ps12892 ps13018 ps12641 ps12646 ps12651 ps12656 ps12662 ps12670 ps12683 ps12705 ps12722 ps12725 ps12747 ps12752 ps12757 ps12763 ps12798 ps12802 ps12810 ps12813 ps12814 ps12815 ps12816 ps12819 ps12834 ps12843 ps12854 ps12885 ps13020 ps12620 ps12640 ps12655 ps12660 ps12666 ps12676 ps12695 ps12702 ps12708 ps12711 ps12713 ps12716 ps12719 ps12721 ps12724 ps12728 ps12731 ps12734 ps12738 ps12742 ps12746 ps12751 ps12756 ps12760 ps12770 ps12775 ps12780 ps12800 ps12804 ps12808 ps12826 ps12831 ps12842 ps12853 ps12863 ps12880 ps12889 ps13021 ps13096 ps12644 ps12652 ps12657 ps12663 ps12672 ps12693 ps12697 ps12706 ps12709 ps12712 ps12715 ps12718 ps12720 ps12723 ps12730 ps12733 ps12737 ps12741 ps12745 ps12750 ps12759 ps12767 ps12772 ps12774 ps12779 ps12799 ps12803 ps12807 ps12821 ps12827 ps12839 ps12847 ps12856 ps12882 ps12894 ps13014 ps13015 ps14196 ps12653 ps12658 ps12664 ps12673 ps12714 ps12717 ps12735 ps12739 ps12743 ps12748 ps12753 ps12758 ps12796 ps12801 ps12805 ps12818 ps12822 ps12829 ps12841 ps12850 ps12861 ps13017 ps13053 ps13054 ps13055 ps13057 ps13058 ps13097 ps13099 ps13101 ps13104 ps13107 ps13108 ps13109 ps13110 ps13111 ps13112 ps13113 ps13114 ps13115 ps13116 ps13117 ps13119 ps13120 ps13121 ps13122 ps13125 ps13126 ps13127 ps13128 ps13129 ps13130 ps13133 ps13201 ps13209 ps13530 ps13531 ps13537 ps13552 ps13561 ps13601 ps13610 ps13631 ps13641 ps13650 ps13706 ps13707 ps13708 ps13710 ps13711 ps13716 ps13717 ps13719 ps13739 ps13753 ps13764 ps13778 ps13786 ps13855 ps13887 ps13888 ps13900 ps13912 ps13921 ps13926 ps13935 ps14003 ps14024 ps14084 ps14140 ps14150 ps14160 ps14164 ps14169 ps14170 ps14171 ps14177 ps14178 ps14191 ps14235

We apologize for the inconvenience and if you have any questions, please contact our support team.

Update 7:20PM: We will beginning the above scheduled maintenance in about 10 minutes. Stay tuned for updates!

Update 8:46PM: Whoops! Forgot to update this post. In any case, all of the mentioned Private Servers should have been up and functioning by 8:00PM. If for any reason you’re experiencing any issues, please contact support.

-The Happy DreamHost High Voltage Robot


8:23 am

some unexpected Private Server downtime

Posted (August 21st, 2009 at 8:23 am PST) by patrickm

We are currently scrambling to troubleshoot and resolve an issue that is causing lots of downtime for some Private Servers.  A detailed list of affected servers is being generated, but we wanted to get this update out, so you know we are working on the problem.

Update: This has been resolved – it turned out to be an issue with a new version of our backup script which we have removed and so far everything appears stable. We’re closing out the related support tickets while verifying that all PS machines complained about are up and running again. We have decided not to list all individually effected ps machines as the list would be rather extensive and would likely just clutter up the posting more than it would be of any real help (our admin team is monitoring all Private Servers so any stragglers that were not running after the fix should have been repaired already).

Update: we found two servers that still had the bad script and have removed it and rebooted them so those two should be back up in the next 5-10 minutes.

Update: after conferring with the author of the new backup script it was determined that the script had been properly vetted (it was already running on several machines without any errors for a week before it was updated everywhere). The problem turned out to be with the method we use for copying the backup data and the admin team is fixing that now so this should not come up again.


2:13 am

email downtime on multiple clusters

Posted (August 21st, 2009 at 2:13 am PST) by patrickm

We are currently experiencing email downtime on the Homie, Janky and Postal clusters of servers.  We are actively working with our Admin team to resolve this issue and hope to have it ironed out quickly.  We sincerely apologize for the inconvenience that this is causing.

You can determine which email cluster you are on from the “Account Status” drop-down menu in your Web Panel.

We will update this post as soon as we have more information, or a resolution!

UPDATE: Still working on this, but wanted to clarify that the ‘sack’ email server is included in this.  Sorry for that omission, as sack is a Postal server, so I should have been more specific.

Update 3:53 AM PDT: It looks like the front-end MySQL server which mail uses was having some issues under the load it was getting. The auth portion of the server was moved to another front-end machine to help alleviate the load and the original one was tweaked to help increase the amount of quiries/sec it could handle. Everything now appears to be good/resolved and we will closely monitor the load on the server for the next couple hours to make sure it doesn’t start having issues again.

Update 5:30 AM PDT: We are aware of the issue with SMTP and have been working to resolve that.  Again, our apologizes over this issue, as all of our tests showed that SMTP was working under this new configuration.  Unless we have to take down the MySQL server to make changes again, you should be able to send mail using WebMail at this time.  More details to follow.

Update 6:30 AM PDT: Once again, we apologize for the continued downtime and the inconvenience that this has caused.  We had to make some more changes to these mail servers to get everything ironed out.  We have pushed out the new configurations to the mail servers and accounts are coming back online now.  Initial testing of these accounts shows that email can be sent and received from both WebMail and Email Software.

This issue will have also caused some delayed email last night and this morning, as Postfix had times that it could not reliably connect to its database.  Those messages are flowing normally now, so there may be a little backlog, but all should be back to normal with that soon otherwise.

We will be closely monitoring all servers involved and will jump on any new issues that arise, though everything is looking great now!  If you are still experiencing any issues with your email accounts, please contact us in Support, so that we can check that out for you.


11:21 am

Server greenlantern being offloaded.

Posted (August 20th, 2009 at 11:21 am PST) by mir

We’ve fixed a number of issues with this server (including dealing with a few customer where were taking up more than their fair share of the server resources) and while that has helped we’re still seeing unacceptable load spikes so we’ve decided to move a portion of the users off of this server (since our checks on apache servers, disk usage and resources consumption are all coming up clean this seems to be the most likely way to resolve the spikes). Moving data can take a while so we’re estimating no more than a week to complete the moves (less if the users selected happen to have moderate space consumption). We’ll update this once that completes.


8:01 am

replacing hardware for a Homie Private Server host

Posted (August 20th, 2009 at 8:01 am PST) by patrickm

We have discovered an issue with one of the Homie Private Server host machines, which requires that we completely replace the hardware.  This has already caused downtime overnight, but we weren’t able to attack this issue until this morning.  We sincerely apologize for the inconvenience that this has caused and continues to cause.

The following Private Servers are affected:
ps13235 ps13252 ps13267 ps13278 ps13290 ps13300 ps13321 ps13330 ps13340 ps13352 ps13361 ps13371 ps13397 ps13407 ps13417 ps13428 ps13450 ps13460 ps13470 ps13487 ps13503 ps13513 ps13524 ps13538 ps13545 ps13555 ps13566 ps13741 ps13760 ps13767 ps13787 ps13832 ps13834 ps13863 ps13864 ps13869 ps13871 ps13879 ps13890 ps13895 ps13906 ps13915 ps13924 ps13932 ps13972 ps13984 ps14010 ps14048 ps14063 ps14072 ps14129 ps14136 ps14139 ps14147 ps14189

This process should take less than an hour to complete and we will update this post when that has finished and all the PS guest servers are running again!

Update – 10:06am (pst): This should be resolved now. All of the PS servers listed above have been moved over and are up and running. If you’re on one of the above servers and are still having issues, please contact support immediately to have it investigated further.


12:00 pm

Avengers, chivas, heat, cavs, falcons and hawks all going down for a moment.

Posted (August 19th, 2009 at 12:00 pm PST) by Leon

Unfortunately, it looks like we have a power strip that is not working correctly so we are replacing it as preventative maintenance. They should only be down for about 15 minutes or so while we replace the strip. This will occur sometime within the next 2 hours.

Update 12:13: All set! Quicker than I thought.


4:17 pm

Mailbox managers are temporarily down

Posted (August 18th, 2009 at 4:17 pm PST) by andrea

We’re taking down the mailbox managers for database upgrades that are needed to fix a current bug. This upgrade should also speed things up on mailboxes.yourdomainname.com when it’s done. Sorry about the inconvenience this is causing! If you have access to the web panel, you can take care of some of the changes you would usually use the mailbox manager for. Unfortunately, this also means you won’t be able to access your messages in the Junk Folder for now, and you won’t be able to edit junk filter settings. Again, for the latter the panel should do as a substitute for now.

We’ll post an update when the mailbox manager is back up.

UPDATE 5:50pm PST: The mailbox manager is up again, thanks for your patience. Please let us know if you run into any issues using it – submit a support ticket through the web panel with details, please.


6:33 pm

Machines getting new hardware!

Posted (August 17th, 2009 at 6:33 pm PST) by kitchen

Due to some recent issues, we will be migrating the following machines over to new hardware:

camaro
cutlass
fiero
horchata
impala
voss
soy
brandine
gaz
salt
bart

We will be starting with camaro and horchata, and continuing from there. Downtime for each machine should be approximately 30 minutes while the process completes. We will post updates as things progress.

Update: 7:56pm pacific: camaro is finished. The new machine horchata was moving to apparently has a faulty nic, so we’ll be moving it again, and we’ll be starting on soy very shortly as well.

Update: 8:00pm pacific: we are also moving impala and voss right now.

Update: 8:21pm pacific: horchata, voss, impala, and soy should all be done at this time. If you are having any issues with these 4 machines, please contact support!

Update: 8:52pm pacific: cutlass and fiero going now.

Update: 9:05pm pacific: cutlass and fiero should be all set!

Update: 8/18/09 3:05pm pacific: Starting a transition for Fidget (not listed above)

Update: 8/18/09 3:27pm pacific: Staring a transition for Ike (not listed above)

Update: 8/18/09 3:56pm pacific: Fidget finished.

Update: 8/18/09 4:13pm pacific: Ike finished.

Update 8/19/09 4:10pm: mountaindew started

Update 8/19/09 5:25pm: coke started

Update 8/19/09 5:40pm: mountaindew finished

Update 8/19/09 5:45pm: brisk started

Update 8/19/09 6:36pm: coke finished

Update 8/19/09 7:18pm: brisk finished


 
© 1996-2009, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
10:28 am

Upgrading hanjin to lenny

Posted (August 17th, 2009 at 10:28 am PST) by kitchen

The machine ‘hanjin’ (also known as backup.dreamhost.com’ is getting upgraded to debian lenny right now. There will be a quick reboot after the upgrade and then another one shortly thereafter to upgrade the kernel. This shouldn’t take very long and will only result in downtime while the machine is being rebooted.

We apologize for any inconvenience this may cause and thank you for your patience!


3:37 pm

Mailman getting upgraded @ 4PM PDT

Posted (August 12th, 2009 at 3:37 pm PST) by beth

We will be upgrading mailman today at 4PM PDT, so discussion lists will experience downtime during this time. The upgrades should not take longer than an hour, and all discussion lists should be fully functional after.


6:06 pm

Upgrading uplinks for 2 racks of servers

Posted (August 10th, 2009 at 6:06 pm PST) by DreamHost

Tomorrow night at 9pm pacific time, August 11, 2009, we will be upgrading the uplinks on two of our racks to be faster and go through a more optimized path through our network. This should only cause a momentary blip in network traffic, but just in case it doesn’t, this is the warning!

bears sox packers cowboys brewers penguins oilers expos phillies eagles wizards hawks falcons cavs heat chivas avengers


10:37 am

Intermittent hosting server outages on our Homie cluster.

Posted (August 8th, 2009 at 10:37 am PST) by mir

A specific subset of our hosting machines have been having intermittent load and connection issues due to an central server which they mount being unresponsive for short periods. One of our admins is in the data center now and will be moving the servers to another mount (a machine which has not been having problems). Here is a list of servers which have been effected:

deathlok
forge
supergirl
twoface
swampthing
angel
mekon
wolfbane
castellano
bonanno
walsh
blade
humantorch
ghostrider
cannonball
silversurfer
namor
wasp
mrfantastic
angel
spawn
greenlantern
shadow
quicksilver
constantine
wolfsbane
dredd
zatara
proty
exodus
preacher
snowbird
professorx
johnnyblaze
fay
rogue
dalitz
thor
apocalypse
avalanche
trafficante

We’ll update this post once the matter is resolved.

Update: The admin team believes this issue is resolved (we found a maintenance script that appears to have been causing it and disabled that).

UPDATE 08/09/09 3pm – Sorry about mis-resolving the ticket, the problem is still present. Our admin team has been working toward resolution. We’ll post more updates as we see things progress. We appreciate your understanding and patience during this period.

UPDATE 08/10/09 11am – The problem with the above servers have been resolved as of last night, but we kept the post going to make sure there are no ‘aftershakes’. If you’re still having any sort of issues with your web server, please don’t hesitate to contact support with the specifics.


9:43 am

Bullseye server down.

Posted (August 8th, 2009 at 9:43 am PST) by mir

We are very sorry for the inconvenience but this machine has not been coming back up properly when we attempt to power cycle it. Fortunately our admin team is on the case and we have someone on site working on the problem. We’ll update this post as soon as we have a resolution.

Update: The server is back up and running – the issue was actually related to this other post:

http://www.dreamhoststatus.com/2009/08/08/intermittant-hosting-server-outages-on-our-homie-cluster/

It was not able to start up again though (unlike the other servers) without the personal attention of one of our admins. We’re keeping an eye on it but it should be fine from here on out.


1:50 am

Unexpected Panel Downtime

Posted (August 6th, 2009 at 1:50 am PST) by patrickm

Our central database server is having some issues, which is causing downtime for the Customer Web Panel at this time.  We are working hard to suss out the issue and resolve the problem as quickly as possible.  We’ll update this post when this issue is resolved.  We sincerely apologize for the inconvenience that this is causing.

Update 8/6 2:20am Pacific –
This issue is now resolved.  We don’t have any additional info on why this happened, but we have begun an investigation to determine that and how to prevent it from happening again in the future.  If we are able to, we will update this post with more information.  If you are still experiencing issues connecting to the Web Panel, please contact Support and they will be happy to help!


6:43 pm

Fiero Lost Its Wheels

Posted (August 5th, 2009 at 6:43 pm PST) by Leon

The fiero webserver is currently being moved to new hardware due to its NIC card going bad. We hope to have it back up soon and we apologize for the inconvenience. Please check back for updates!

Update 7:58PM: Sorry that it took a bit longer than normal but the server should be back up and functioning as normal. If you’re still experiencing issues, please contact our support team!

-The Happy DreamHost “Sports Car” Robot


6:09 pm

Bonanno Problems Today

Posted (July 31st, 2009 at 6:09 pm PST) by Ralph

Sorry to those of you on the machine Bonanno who have experienced problems today. Your problems are similar to those experienced by users of the machine Barker. Both machines have been getting upgrade to more stable hardware. The problem has come up with our data migration tool. This tool had previously not caused us any problems but in this go around with upgrades it has caused a high load and stability problems. As with Barker we have upgraded the ram and dealt with any resource abusive processes. The machine is stable now and we expect the data migration to be finished early next week.

UPDATE 8/3/09 11:45am: The migration to a more stable server has completed, and the server was also given a RAM upgrade. We found some heavy users as well, and are preparing to offload some customers to further stabilize the machine. Due to all these changes and processes, the machine was getting a bit loady, so we just rebooted it – it’s back up again. We’re keeping an eye on it. The upcoming offload may cause some network saturation, so a bit higher than normal load is expected, but this should all be over soon. We appreciate your patience with this process.

UPDATE 8/3/09 1:27pm: We’re looking into the recent high load on the machine that is causing intermittent issues such as timeouts and internal server errors. Sorry about the continued inconvenience. We’ll update this post when we have more details.

UPDATE 8/4/09 11am: The load you’ve been seeing throughout yesterday and today is apparently still caused by the number of customers on the machine, which we are already alleviating with moving off a good chunk of them. We do have a limit on the number of accounts we add to a machine, but we don’t limit the number of users within the amount, so at times it can grow out of proportion. We have new machines going up, and the offload for your server is almost halfway done. We expect the stability to return within days now. We appreciate your patience during this process. We’ll post updates as we go.

UPDATE 8/5/09 1:40pm PST: The offload has finished, and other than a few smallish spikes here and there, the machine has been stable today. We are continuing the monitoring, but for now we consider this case closed. If you’re still experiencing issues with your domains on this machine, please contact support with the specifics, so we can look into it for you.


10:07 am

Continued problems with Barker

Posted (July 31st, 2009 at 10:07 am PST) by Ralph

Some of you have been keeping up with the problems with the hosting machine Barker. After we upgraded the hardware for it we had to run data migrations that were causing the server to experience a high load and provide intermittent service. Last night we upgraded the ram on the machine and it appeared that the machine was going to be fine. Unfortunately this morning the high load issues have returned. We are actively investigating this issue and will report here with more information as soon as we have it. Sorry about the problems this has caused you. Previous status reports on this incident were incorrectly included in a previous post about the upgrade. Our further apologies for not separating the status reports.

Update 12:47 Pacific: There was a resource abusive user affecting the server. The user has been contacted and their processes have been stopped. The machine has been working fine since then. This ticket however will not be closed until the migrations are complete and I am 100% certain the issues are resolved. I thank you all for you patience since this matter has gone on since after the upgrade was complete.


3:00 pm

Luthor server having problems

Posted (July 30th, 2009 at 3:00 pm PST) by danwy

Our luthor server has been having some intermittent issues for the past few days. We’ve been unable to find anything specific on the server causing these loads, so our admins suspect that it’s resources are just being too heavily used. We’re in the process of offloading 300 accounts from this server onto a new one in hopes of freeing up some resources and returning stability to the server. We’re expecting the offloads to take about 24 hours and we will re-evaluate the server after that time to determine if any more work needs to be done on it. We’re sorry for any inconvenience this causes you.

Update 7/31 14:10 Pacific: The offloading has completed and there was also further maintenance done on the machine. It seems to be working optimally at this time. We will continue to monitor, but it seems like Luthor is going to be fine now.


 
© 1996-2009, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
11:27 am

Joomla Advanced One-Click Updated to 1.5.13 — Security Fixes!

Posted (July 28th, 2009 at 11:27 am PST) by Karl

The one-click installer has been updated for Joomla 1.5.13. This release contains “26 bug fixes, two moderate-level security fixes and one low-level security fix”, so please update as soon as possible!

Release info at Joomla’s site, here.


5:54 pm

Waldorf’s gone over the balcony

Posted (July 27th, 2009 at 5:54 pm PST) by Jason C

Due to a drive issue with the network file server waldorf — which serves folks on the spacey cluster — users on the following servers may be seeing slowdowns and inaccessible sites:

amp arizona barqs bawls boba brisk calpico cappuccino caprisun carlosrobles chai clamato codered coffee coke crush crystallight culligan dasani dietrite eggnog evian fiji frappe fresca fuze geyser gingerale hansens hiball ibc inko izze jolt koolaid lingolook lipton martinellis minutemaid monster mountaindew mrpibb mug musclemilk nesbitt nestea orangebang ovaltine pellegrino powerade propel refresco riptide rockstar rootbeer schweppes seltzer shasta slice snapple sparkletts squirt stewarts sunkist sunnyd tab tampico tang tea tonic tropicana v8 welchs yoohoo

Our admin team is working on resolving this now and this entry will be updated as the issue progresses.

Update: We’ve managed to get waldorf back under control. The servers on the list above are all stabilizing and customer sites should be coming up again. If you’re on one of the servers listed above and find that to not be the case, please don’t hesitate to contact support.


5:53 pm

Server migrations coming in August

Posted (July 27th, 2009 at 5:53 pm PST) by andrea

If your server is one of the following, you should notice improved stability and performance soon. We’re migrating them to new hardware during the next month. You won’t be receiving any more notices, as other than a possible brief outage (about 15 mins) while the switch happens, the data migration should be transparent to you. This is just heads up that if you’ve been experiencing troubles with your sites, rest assured, they are being worked on. You can check for your web server name under ‘Account Status’ in the upper right corner.
If you need help in the meantime (even if you’re on these servers) please contact support through the web panel, so they can assist you.

castiglia
gambino
lanksy
luciano
rothstein
scalise
schultz
mcgurn
mangano
bonanno


3:41 pm

Barker and Dalitz being migrated to new hardware

Posted (July 27th, 2009 at 3:41 pm PST) by DreamHost

Barker and Dalitz are being moved to new hardware to help deal with performance and stability issues. The total downtime should be around 15 minutes. This will affect web, shell, file, and jabber services. MySQL and email should not be affected.

These are back online!

Update 7/30/09 11:25am PST: Even though the server was moved to new hardware and started working as we noted above shortly after, barker’s data migration is still going. We anticipate it to finish soon, please check back for updates. The migration is causing some network saturation which may result in intermittent service. We apologize for the continued inconvenience.

Update 7/30/09 15:20 Pacific: We upgraded the RAM on the machine barker and it is handling it’s data migrations much nicer. We will continue to monitor the machine through it’s migrations but it seems like we are out of the major stability issues with the machine.

Update 7/31/09 10:12 Pacific:Unfortunately this morning, barkers problems continued. We are currently investigating and the status post about this issue has been moved here:

http://www.dreamhoststatus.com/2009/07/31/continued-problems-with-barker/


3:39 pm

Homie Networking Issues (And An Apology)

Posted (July 27th, 2009 at 3:39 pm PST) by Jason C

A large number of our customers may have noticed issues with their servers on the homie cluster about 4 hours prior to the time of this posting. It turns out there were severe networking issues that affected a number of racks in one of our datacenters. Because of this, there was some noticeable downtime for more than a few people. From my understanding, this matter should now be resolved for most — if not all — of the customers it was affecting.

Apparently, there is was some minor fallout on a handful of servers, but our Network Operations techs have been making sure that everyone is back up and plugging along without any issues.

Normally we prefer to have a post up detailing what is going on as the issue happen. In this case, our team had to scramble so fast to take care of the issue that a post here was the last thing we thought about. Furthermore, the point person on large scale support issues for this week (which is me) wasn’t receiving notifications from our system. Had the messages come thru, a post would have been up almost immediately.

From a support standpoint, we see this as unacceptable and we apologize profusely to the people who had no idea what was going on with their sites.  Hopefully, an issue like this won’t occur again — but if it does, please know that we will do our best to keep all of our customers up to date thru this channel at the very least.


1:11 pm

Delayed mail updates on our Homie cluster.

Posted (July 25th, 2009 at 1:11 pm PST) by mir

Due to an internal user (machine user not a live person less there be any concern on that front) mysteriously changing status and causing user configurations to hang customers have been noticing delays in updating and creating mail users and addresses. We have found and fixed the issue so that updates are running (this was taking care of some hours ago and so far the items we have checked have completed successfully now) but our development team is still working on what causes the status change. Everything should have processed normally in the meantime and we’ll be keeping an eye on that user to ensure that this does not cause any more noticeable problems.


2:14 am

castellano is down for a fsck

Posted (July 25th, 2009 at 2:14 am PST) by Sandon

Castellano was unable to mount one of its partitions due to some file-system errors so I am running an fsck (file-system check) on it right now. Customer data will be in-accessible until this completes.

The ETA is 1-4 hours depending on the amount of data on the file-system. More updates to come as they are available.

Update 08:05 AM PDT It looks like the file-system check had to restart from the beginning after the first (long) pass. It completed another first pass without any errors and is now on its second pass. This usually means its about 75% done.

Update 11:55 AM PDT This took longer than expected to complete but it has now finished and I am rebooting the server which should only take a few minutes (everything should come up properly at that time).

Update 12:15 PM PDT I believe everything is in working order (one apache did not come up properly but I fixed that) so this issue should now be resolved (I’ll be testing cases as I close out the support related to this matter so that we can be certain).

Update 12:40 PM PDT So far almost all sites are coming up just fine but in a few cases I have had to run a config of the domain apache service to get an httpd.conf error to go away. Just to be thorough I am running those for every apache (that should take around 30 minutes) so if you are receiving that error it should go away by around 1 PM PDT.


2:09 pm

Server Cupcake having its hardware replaced

Posted (July 24th, 2009 at 2:09 pm PST) by Ralph

We are currently replacing the hardware on cupcake due to a misconfiguration issue. As soon as we have the machine replaced we will update here. Sorry about the downtime.

Update 2:28 PDT Server is back up without an issues. If you notice any problems please contact support.


6:53 am

file ownership permission issues

Posted (July 24th, 2009 at 6:53 am PST) by patrickm

I am sorry to report that a configuration issue has caused a change to the ownership permissions of many user’s Home directories on the servers.  This appears to mostly affect the Homie cluster, but may not be limited to that.

We are actively addressing this issue, which involves running some configuration scripts on the servers.  This is taking some time to complete, as we are working with each server to make sure this is done correctly.

This issue will cause errors when making changes to files on the server and may also cause errors to display on sites, but not in every case.  We understand this is a problem for our customers and are working as quickly as possible to get them all fixed up.

We have no estimate on when this will be finished, but will update as often as possible with our progress!  If you are unsure if you are affected by this issue, please contact support and if this issue is present, they can manually fix your account very quickly.

UPDATE 9:00am Pacific: We are still working to resolve this issue with all the Homie servers.  As requested in this post’s comments, here’s what happened:

We removed more than 12,000 duplicate UIDs from the system yesterday.  This required that an update be pushed out to set the current/correct UIDs for FTP/Shell users.  Unfortunately, that update was not completed and it left these users in their current state.

You can identify this issue by looking at your Home directory on the server.  If the owner of the files under your account is a string of numbers instead of your username, you are affected by this issue.

Again, please feel free to contact support and they will quickly resolve your issue, but we are pushing out this update across all Homie servers at this time.  So, this issue will be resolved when those updates complete.


2:48 pm

New Rails version 2.3.3!

Posted (July 23rd, 2009 at 2:48 pm PST) by Terri

We have the packages built and ready to go! During the next week, we’ll slowly be rolling out Rails 2.3.3. As usual, to be safe, we’re recommending to our users that they freeze their Rails applications. This is really easy to do! All you have to do is login via to your shell account, change directories into your app, and run:

rake rails:freeze:gems

All of our servers should be updated by the end of next week. If you see any problems with your installs related to the upgrade, please contact support.


 
© 1996-2009, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
12:36 am

homie-vserver38 down for a fsck

Posted (July 23rd, 2009 at 12:36 am PST) by Sandon

It looks like the PS server host ‘homie-vserver38′ was getting some file-system errors so I had to take it down to run a fsck. This is affecting people on the following PS’s:

ps7789, ps7791, ps7792, ps7793, ps7794, ps7795, ps7796, ps7797, ps7798, ps7800, ps7801, ps7802, ps7804, ps7806, ps7807, ps7808, ps7809, ps7810, ps7812, ps7813, ps7814, ps7815, ps7816, ps7817, ps7818, ps7819, ps7820, ps7821, ps7823, ps7824, ps7832, ps7835, ps7870, ps7873, ps7874, ps7876, ps7878, ps7886, ps8022, ps8029, ps8052, ps8065, ps8066, ps8078, ps8079, ps8080, ps8081, ps8083, ps8084, ps8283, ps8372

The ETA is 1-2 hours but in some rare cases an fsck can go much quicker or take much longer than this. More updates to come as they are available.

Update: 02:03 AM PDT: It looks like this is taking a bit on the long side. Unfortunately there is no way to speed it up and it will take as long as it takes. Usually even the worst case fsck does not take more than 4 hours.

Update: 02:20 AM PDT: It has finished its initial passes which usually means its about 70-80% done but impossible to know the exact percentage.

Update: 04:11 AM PDT: It looks like its still going but on the final pass now which usually means 90%+ done. The fsck speed is usually directly proportional to the amount of files on the file-system and it looks like this one has more than usual. I apologize for the delay in getting the machine back up and running.

Update: 04:38 AM PDT: Ok, fsck has finished and the machine has been rebooted and all guests are back up. If you are on any of the above servers and still having issues please contact support.


5:48 pm

CANCELLED: randy/janky getting reorganized slightly!

Posted (July 21st, 2009 at 5:48 pm PST) by kitchen

Update 5:38pm pacific on July 22nd: We have actually taken a step back and looked and realized we don’t actually need to move any of these servers, so we’re cancelling this maintenance window. Sorry for the alarm!

Monday night / tuesday morning, July 27th – 28th, we’ll be doing a little bit of reorganizing of hardware in the randy and janky clusters. We need to move around a few file servers, which involves turning them off, so there will be some downtime for a portion of the randy and janky clusters. We estimate that this process will take no longer than 2-4 hours, and we will begin around midnight pacific time. We will put up a post a few hours before we begin and will continue to update it as things progress.

We thank you for your patience and apologize in advance for any inconveniences!

Sincerely,
The Happy DreamHost Wee Hours Cluster Organizer Team


9:07 am

WebMail Downtime

Posted (July 21st, 2009 at 9:07 am PST) by brians

We are fully aware of the webmail problems. Our administration team is currently investigating the issue. Everything will be resolved shortly. We sincerely apologize for the inconvenience this has caused and appreciate your continued patience. Please check back for future updates.

UPDATE 09:41 PM PST The issue has been resolved. This recent episode is separate from the incident over the weekend. This mornings downtime was caused by a hung query on one of central databases. Restarting mysql appears to have resolved the issue. We will continue to monitor the stability of the WebMail service. Again we apologize for any inconvenience that this has caused. If you are still experiencing issues, please submit a ticket through our support system.


10:07 am

Homie Machines Getting RAM Upgrade

Posted (July 20th, 2009 at 10:07 am PST) by Leon

I will be upgrading memory in the following machines: wonderwoman, mangano, wasp, penguin, capone, rothstein & avalanche.

Downtime should not be greater than about 5-10 minutes per machine and when it’s done, we should (hopefully!) be seeing better performance from the listed machines.

Edit (11:00am): Aaannnnddddd done. Sort of. Ran out of RAM so mangano and rothstein will be upgraded later.


9:16 am

Webmail problems

Posted (July 18th, 2009 at 9:16 am PST) by Ralph

We are currently investigating webmail problems that are causing your webmail sites to show php errors. We are actively working on this right now and I will update this post as soon as we have more information.

Update6:45pm Pacific time – Problem is resolved. We found the problem where php wasn’t being processed correctly and we resolved the issue. Webmail should be working without any problems now. Sorry about the interruption in service.

Update 10:40 PM PDT It looks like some issues are cropping up again. An admin is investigating the cause.

Update 12:05 PM PDT This issue should be resolved as of about 30 minutes ago. One of the webmail machines had some issues which was intermittent problems due to the fact that it was only one of the several webmail servers we have behind a load balancer.

Update 9:50 AM PDT 7-18 We are experiencing further problems with webmail and PEAR errors. We are investigating. Sorry about the continued trouble with webmail.

Update 10:34 AM PDT We’ve narrowed the issue down to a specific server which we have taken out of the rotation (webmail runs on multiple machines) while our admin team is investigating. This should take care of the errors (we won’t return that machine to the rotation until the matter is resolved).

Update 11:50 AM PDT So it looks like in an effort to fix some of the issues that specific mail server was exhibiting we ran a config on it (unfortunately this was a bad idea as it upgraded things that should have remained as they were including the PHP version which resulted in the PEAR database errors that were coming up intermittently when customers hit that mail server). We’re leaving it out of the rotation until Monday when we can make sure it’s fully fixed and re-integrate it (so there should be no more errors caused by this situation).


3:22 am

Network problems due to Distribution Switch

Posted (July 18th, 2009 at 3:22 am PST) by Sandon

it looks like one of our distribution switches in our primary data center became unresponsive and all racks/switches attached to it have gone offline. This is affecting the following machines:

abe, abraxor, accordionguy, accserver, adamserver, adbo, adonis, agamemnon, ajax, alc-tftp, ampedsports, amw, annie, apezone, aphrodite, apollo, apu, arda, aries, arnie, atlas, atra, avenus, azcuehnos, baachus, balto, barriodelcarmen, bart, bashtage, batserver, bbd, ben, berniemedia, betelgeuse, bettyboop, bic, biffster, bimbodulcop, bitters, blackfalcon-web, blackfday, blackhole, blatant, blizzard, blogshare, bloodcopy, blueprint, bobo, bogus, bollocks, bomberstudios, booger, borealis, brandine, brawndo, breckgear, brockman, browser, broztrain, burns, caiman, caitlin, calypso, caparazon, carl, casaclubtv, casper, celeb, celtics, cfndc, chalmers, charon, charybdis, checkers, cheech, cheeky, cheritek, chronos, chubu, ciclo22, clancy, cletus, concorde, coolblue, cupid, da, daedalus, daffy, damage, danton, dawber, dewey, dib, dione, dionysus, diversity, djm, djpreet, donald, dragonball, drederick, drooper, duffman, dusqi, dynomutt, ecards, eclipse, ed, eduagora, einstein, electra, elmyra, elvis, emanuel, emg, engine57, enigmarey, erebus, erida, eros, evildead, evolve, ewaste1, ewok, farnsworth, felham, fiercee, flamjam, flanders, flowstudios, forkpile, fresh, fxdollar, ga, gassi1, gaz, getrightmusic, ghost, gimli, gir, girl, glass, gloop, gnif, gobehost, gorgon, gotcha, grandstand, grasshopper, grimbald, grimm, gromit, guava, gumoz, hades, hadron, harpo, haystack, hb02, hdotnet, hecate, hector, helios, hephaestus, hera, heracles, hermes, hestia, hibbert, hmtotc, homer, huey, hypatia, hyperblocks, hypnos, i2, iandi, icarus, iceandrock, ichigo, identity, iinst1, imu, insie, instantcenter, iotatec, iris, itauna, jankymail-a3, janky-vserver01, janky-vserver02, janky-vserver03, janky-vserver04, janky-vserver05, janky-vserver06, janky-vserver07, janky-vserver08, janky-vserver09, janky-vserver10, janky-vserver11, janky-vserver12, janky-vserver13, janky-vserver14, janky-vserver15, janky-vserver16, janky-vserver17, janky-vserver18, janky-vserver19, janky-vserver20, janky-vsql01, janky-vsql02, janky-vsql03, jarcas, jasper, jbutton, jebediah, jelly, jezebel, jiborrabotia, jimbo, jingoro, jove, jtse, jubjub, kaysnature, kearny, keei, kesido, keyiflimutfak, keysersoze, kiley, kitwe, kongerweb1, kopi, krabappel, kratos, kronos, krusty, kztixo, lablogs, lagrange, larry, leancode, leichtman, lenny, leto, limbo-randy, linkmoney, linneo, lisa, lolicg, lovejoy, lucy, madotter, madsamurai, managedq, manjula, marbles, marcdorcel, marge, marhgil, maya, mccallister, mcclure, mediadirect, medusa, mho, midas, mikaeljansson, millhouse, mills, minerva, misfit, mita, mll, mona, monkey, montecristo, mooshi, mooze, morpheus, mtrends, mudpuppy, mundodigital, muttley, mwl, mysportonline, nanook, ned, nelson, nemesis, netrino, netseyredin, nipper, noodledee, noticiaria, numbers, oden, odysseus, odyssey, oedipus, ohso, oneclick1, oneclick2, optimusprime, oriza, oscarfishlover, osiris, osteele, otto, pan, pandora, patty, pegasus, persephone, phan, phi, pimp, plubble, pmbd, pogo, popehat, porker, poseidon, postal-vserver01, postal-vserver02, postal-vserver03, postal-vserver04, postal-vserver05, postal-vserver06, postal-vserver07, postal-vserver08, postal-vserver09, postal-vserver10, postal-vserver11, postal-vserver12, postal-vserver13, postal-vsql01, precious, prospero, proteus, ps3626, ps3710, ps3845, ps3865, ps3900, ps3927, ps3930, ps3953, ps3957, ps3982, ps3997, ps4053, ps4072, ps4087, ps4098, ps4131, ps4168, ps4206, ps4326, ps4337, ps4419, ps4421, ps4473, ps4571, ps4582, ps4583, ps4693, ps4721, ps4729, ps4730, ps4731, ps4978, ps5037, ps5120, ps5230, ps5244, ps5251, ps5257, ps5263, ps5291, ps5389, ps5524, ps5548, ps5594, ps5608, ps5633, ps5635, ps5694, ps5724, ps5741, ps5752, ps5755, ps5760, ps5826, ps6152, ps6272, ps6300, ps6366, ps6417, ps6428, ps6429, ps6432, ps6445, ps6449, ps6457, ps6459, ps6465, ps6467, ps6511, ps6803, ps6848, ps6917, ps7004, ps7013, ps7035, ps7158, ps7283, ps7290, ps7368, ps7371, ps7524, ps7922, ps7938, ps7990, psmysql1829, psmysql1861, psmysql2529, psmysql2532, psmysql2535, psmysql2552, psmysql2553, psmysql2561, psmysql2565, psmysql2570, psmysql2571, psmysql2593, psmysql2596, psmysql2607, psmysql2614, psmysql2634, psmysql2637, psmysql2652, psmysql2844, psmysql2848, psmysql2874, psmysql2997, psmysql3002, psmysql3003, psmysql3011, psmysql3165, psmysql3171, psmysql3228, psmysql3356, psmysql3369, psmysql3516, psmysql3544, psmysql3730, psmysql3811, psmysql3866, psmysql4039, psmysql4149, psmysql4211, psmysql4240, psmysql4247, psmysql4588, psmysql4647, psmysql4655, psmysql4683, psmysql4684, psmysql4881, psmysql4975, psmysql5183, psmysql5278, psmysql5377, psmysql5598, psmysql5756, psmysql5757, psmysql5769, psmysql6082, psmysql6127, psmysql6423, psmysql6433, psmysql6441, psmysql6455, psmysql6526, psmysql6598, psmysql6996, psmysql7040, psmysql7156, psmysql7157, psmysql7360, psmysql745, psmysql7946, psmysql799, psmysql802, psmysql961, psmysql962, psmysql963, psmysql965, psmysql966, psmysql967, psmysql969, psmysql972, psmysql979, psmysql986, psmysql991, puch, quifflet, quimby, quimbys, radiograffiti, ragland, raiden, ralphie, razor, rhea, rhinoco, rhythm, sammi, sanjay, sarabi, schadenfreude, schnoodle, seamus, sempei, seven75, shaneandpeter, shemp, shyfry, si, simflight, siminteractive, sisyphus, sitesablaze, sixtytwo, skimbleshanks, skinner, slinky, smithers, socket, softwa, somaticvision, sowired, spunky-vserver20, spunky-vserver23, ssiws1, streaky, sturman, sturmgroup, sunnywood, svetluska, sweetdreams, syscount, tachibana, tailfin, tak, talkbox, tamiyaclubmovies, tammany, tartarus, tethys, tfotw, thanatos, thetis, tic, tinyblip, tnmedia, tonni, tooth, touchofdeath, touchofdeathforums, triage, triton, tweety, twunbbs, tyche, universoprivate, unusedvps3093, unusedvps471, unusedvps472, unusedvps473, uwtsweb, vidda, vivartes, vivenet, vpg, vrt7, vulcan, wally, warppipevps, webben, weblogstematicos, webmail5, webprojnet, webxl, wiggum, willie, willimedia, wittenberg, wolkanca, woodstock, wyld, xgun, xiondesigns, yerba, yestoronto, yggsdrasil, zdki, zentastic, zero, zeus, zim

We have already dispatched someone to check it out and hopefully will have status updates soon.

Update 05:27 AM PDT After some troubleshooting it looks like there is a hardware problem with the switch and we are replacing it with a spare.

Update 06:13 AM PDT Switch has been replaced and connections are being brought up. Some of the machines on the list should already be responding. The rest should be back up soon.

Update 06:32 AM PDT Networking is back on all the machines. There are just a couple of stragglers that might need to be poked due to losing the network connection with their file-server but almost everything should be back to normal now.

Update 07:37 AM PDT It looks like there is a problem with some of the uplinks on the switch which is causing some high loads and slow speeds. We are still working on resolving this issue.

Update 08:14 AM PDT During a configuration update to fix the uplink issue it looks like the switch has gone down again so the above servers are down yet again. Our apologies. It should be back up much quicker than last time.

Update 08:42 AM PDT Switch is back up and the previous issue is solved so things should stay stable now. A couple servers are going to need some attention but most of the servers should be back up again.

Update 09:08 AM PDT Everything is looking good and it looks like all the machines affected by this have fully recovered and are at normal loads/speed.


7:24 pm

Avalanche Getting a New Raid Card

Posted (July 16th, 2009 at 7:24 pm PST) by Leon

At around 9:00PM PST today (07/16/2009), we will be swapping out avalanche’s raid card. We’ve recently noticed a few issues with it and to prevent any further issues with it we will be swapping it out with a fresh one. We estimate the downtime to be around 15 minutes but hopefully it should be around 5 minutes. Please check back for updates!

Update 09:25PM PST: We took it down a little later than the time posted but the overall downtime for avalanche was only about 10 minutes. If you’re experiencing any issues, please contact our support team!

-The Happy DreamHost Raid Fixin’ Robot!


8:12 am

Forced Password Changes (to enhance security)

Posted (July 16th, 2009 at 8:12 am PST) by David

Some of you may have received several e-mails lately regarding having FTP/shell or e-mail accounts that have very bad, easily guessed or insecure passwords (like the word “password”).

While we’ve attempted to contact our customers and give them plenty of time to deal with these accounts and to change the passwords to something more secure some of them have not responded and we’ve been forced to manually change their passwords for them. This is all done in an effort to improve the overall security of our systems to protect all of our customers from hacking and other abuses. While the overall security of the servers are locked down as tight as we can make them while still providing service, customers account passwords are the most vulnerable entry points to our system.

Changes have been made to our control panel that now prohibit customers from choosing inherently insecure passwords (like “password”, “123456” or account usernames, etc.) however we’ve found that many customers still do have insecure passwords.

If you’ve recently received an e-mail (Subject: DreamHost Security Notice: Insecure Passwords Changed!) that means after many notifications your WebID still had insecure passwords and we were forced to change them for you. The e-mail will lists the FTP/shell and e-mail accounts whose passwords have been changed.

Please log into our control panel and change the passwords for those accounts immediately and give your users their new (secure) passwords. We would strongly recommend that you use the “Pick a new password for me” option as our system will generate very secure random passwords for your accounts.

We apologize for having to resort to this procedure which may cause problems for your users to access their accounts temporarily. However the benefits in increased security for ALL of our customers far outweigh the minor inconvenience for the relatively few customer accounts that are involved.

— The DreamHost We’re Beefing Up Security (for you) Team


8:13 pm

Lights out on greenlantern

Posted (July 14th, 2009 at 8:13 pm PST) by Jason C

Users on the server greenlantern will be seeing issues due to a hardware failure. One of our admins is currently swapping the drives to new hardware and hopefully that will get things back up quickly — but there could be further issues. If things decide to get more complicated, we’ll be sure to update this post and let you know right away.

Update: Apparently, a drive swap isn’t going to bring greenlantern back up quickly. We’re now scrambling to restore the server and copy recent backups onto the new machine.

Update: 10:56 PM PST It looks like there were some complications with the restore process so the machine is still down. One of our admins is now en-route to the data center to work on getting everything back up on the server but there is currently no ETA.

Update: 1:41 AM PST Ok it looks like we were able to get the OS re-installed on the original hardware and the volume with all customer data was completely intact so the machine should be back up (within 20-40 minutes) with no loss of data. I will update this post when everything is back up 100%.

Update: 2:23 AM PST All services have been restored on this server and there has been no visible data-loss. If you are still having any issues and are on ‘greenlantern’ issues please contact support.


11:54 am

DDoS attack on some of our servers

Posted (July 12th, 2009 at 11:54 am PST) by danwy

We’re currently experiencing issues due to a DDoS attack targeting some of our servers. This is resulting in some of our customers sites being unavailable. We’re unsure at this time of exactly what is being affected, but our admin team is currently investigating it and will have more details shortly. We’ll update this post as soon as we have any more detailed information.

UPDATE – 2:45pm (PST): This should have been resolved for the last hour or so now. We just wanted to make sure everything stayed stable before we updated this post. It looks like it was only affecting the redbull server and was due to some ZenCart hacks. We’ve given the affected apache’s new IP’s and that seems to have cleared everything up. If you have any further issues, please don’t hesitate to contact support.


 
© 1996-2009, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
10:37 am

Downtime on DrPepper

Posted (July 8th, 2009 at 10:37 am PST) by brians

The NIC on the machine appears to be faulty. We are in the process of moving the machine to new hardware. The machine will be back online shortly. We sincerely apologize for the inconvenience this has caused and appreciate your patience. Please check back for future updates.

UPDATE: The machine is up and running. Our administration team will continue to actively monitor. Again, we apologize for the inconvenience this has caused. If you are still experiencing stability issues, please submit a support request.


6:32 am

Homie-vserver32 booting issue being looked at

Posted (July 5th, 2009 at 6:32 am PST) by Justin@DH

One of our DreamHost PS host machine for the following PS’s:

ps7051 ps7073 ps7080 ps7087 ps7093 ps7095 ps7096 ps7097 ps7146 ps7183 ps7208 ps7209 ps7345 ps7355 ps7356 ps7403 ps7424 ps7429 ps7473 ps7477 ps7734 ps7735 ps7736 ps7737 ps7738 ps7739 ps7763 ps7779 ps7847 ps7865 ps8035 ps8056 ps8159 ps8160 ps8161 ps8162 ps8167 ps8202 ps8220 ps8252 ps8254 ps8741

Is having trouble loading its special kernel. One of our admins is working on it right now, when we have further info we will update this post. Our apologies for the downtime related to these specific PS’s. We are doing all we can to minimize the downtime.

— UPDATE —

I just heard back from our admin that the issue has been resolved and service should be restored. If you still have problems though please contact support and we would be happy to assist you with any issues.


8:19 pm

Tyche Moving to New Hardware

Posted (July 3rd, 2009 at 8:19 pm PST) by Leon

The web server, tyche, is currently having issues on boot. We’re currently working on moving it to new hardware and should have it back up within the next hour. Check back for updates!

Update 9:10PM: Tyche has now been moved over to new hardware and is happily serving your websites again! If you’re still experiencing issues, please contact our support team.

-The Happy DreamHost Server Moving Robot


11:25 pm

Spunky-vserver 21 is down for the count.

Posted (July 2nd, 2009 at 11:25 pm PST) by Sandon

It looks like the machine is completely unresponsive and we are unable to power-cycle it remotely. We have an admin en-route to the data center to check it out.

It looks like this issue is affecting the following ps guests:

blacksun, denarquivista, heyes, mkb, ps4580, ps5118, ps6258, sugarsmacks

no ETA at the moment but unless something is wrong with the hardware (which is possible) it will probably be about an hour until these are back up.

Update 12:59 AM PST: The host is back up and all its guests are running again. Please contact support if you are any of the above guests and still having issues.


2:21 pm

Fitzpatrick Initializing

Posted (July 1st, 2009 at 2:21 pm PST) by Kyle

Fitzpatrick has been well, on the fritz today and it seems to be due to the RAID array wanting to start initializing itself. We don’t have a ETA yet on when it will finish and unfortunately there isn’t a way to coax it into finishing more expediciously. We’ll provide an update when it is complete, unfortunately the card in this device doesn’t provide a % of completetion so we won’t know quite how much longer until it’s done.

Update 07/05/09 6:09PM: Fitzpatrick has finished initializing and should be good to go. If you have any questions, please contact support.


1:18 pm

Joomla One-Click Updated to 1.5.12 — Security Fixes!

Posted (July 1st, 2009 at 1:18 pm PST) by Karl

The one-click installer has been updated for Joomla 1.5.12. This release contains “a number of bug fixes and three moderate-level security fixes”, so please update as soon as possible!

Release info:
http://www.joomla.org/announcements/release-news/5242-joomla-1512-released.html


12:45 pm

Quick network upgrade tonight! (Completed!)

Posted (July 1st, 2009 at 12:45 pm PST) by DreamHost

Tonight at 10pm Pacific we will be converting a cross-datacenter link from a 10gig link to a 20gig port channel. There likely will be no noticeable impact of this change, but we wanted to give you a heads up!

Edit: This has been completed. Traffic looks normal across the newly beefier link! Please contact support if you are having any problems.


9:40 am

Multiple Cluster Email Authentiction issues

Posted (July 1st, 2009 at 9:40 am PST) by danwy

We’re currently having some issues with our email authentication system. This is causing mail login for many of our customers to not work correctly. Our admin team is aware of this and are already working on restoring service. This should be back up and running shortly. Please keep an eye on this post, as we’ll update it when the issue is resolved. We apologize for any inconvenience this causes you.

Update – 9:55am PST: This issue should be resolved now. We’ve restarted the individual cluster mail services and things seem to be running smoothly now. We’ve tested several of the affected mailboxes and have verified that they are working. If you continue to have issues, it was likely not related to this incident and you should contact support for further assistance.


5:06 am

Morris was Zack Attacked!

Posted (June 28th, 2009 at 5:06 am PST) by Leon

The MySQL server morris is currently having issues during boot up. We’re currently working on moving it to new hardware and should be back up within an hour.

Update 5:26am: We have moved morris onto new hardware and it should be back up. If you’re still experiencing issues, please contact support!

– The Happy DreamHost Early Morning Server Fixin’ Robot


1:22 pm

Falco Getting New Raid Card

Posted (June 27th, 2009 at 1:22 pm PST) by mir

Our admin team is replacing that hardware – it should be back up in the next half hour or so and we’ll update this post as soon as that is the case.

Thank you for your patience!

Update 1:49PM: Falco is now back up and all better!


 
© 1996-2009, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
12:25 pm

Terranova getting a break.

Posted (June 24th, 2009 at 12:25 pm PST) by mir

It looks like this server may not have an optimal distribution of usage (meaning it seem to be getting hit with a lot of usage at the same time which keeps spiking the load) so we are moving some customers off to help with that. Please be patient with this process as the moves will also add to the server load (but once we have completed the process over the next week or so the performance should improve and we’ll be watching to make sure that it does).

Thanks!


2:22 am

avalanche has fallen and can’t get up.

Posted (June 20th, 2009 at 2:22 am PST) by Sandon

It looks like the web-server ‘avalanche’ was unresponsive and did not come back up after being remotely power-cycled. We have dispatched someone to the data center to check out what the problem is.

Update: 02:39 AM PST: The issue has been resolved and the server is booting now. it should be complete back up within 10 minutes.

Update: 02:45 AM PST: Ok, All services on this machine should be 100% restored. If you are on avalanche and still having any issues please contact support.


12:43 pm

Power problems affecting Gambino, Bruno and DreamhostPS machines

Posted (June 19th, 2009 at 12:43 pm PST) by Leon

DreamhostPS affected:

ps10079 ps10688 ps11564 ps8662 ps8677 ps8700 ps8725 ps8871 ps8881 ps8929 ps9041 ps9144 ps9590 ps10080 ps10891 ps11652 ps8665 ps8678 ps8701 ps8726 ps8872 ps8891 ps8935 ps9045 ps9149 ps9674 ps10213 ps10902 ps8650 ps8666 ps8683 ps8703 ps8728 ps8873 ps8908 ps8960 ps9097 ps9161 ps9813 ps10222 ps11154 ps8659 ps8668 ps8684 ps8714 ps8773 ps8876 ps8920 ps8963 ps9135 ps9168 ps9840 ps10367 ps11165 ps8660 ps8674 ps8690 ps8715 ps8774 ps8878 ps8926 ps9034 ps9138 ps9580 ps10582 ps11174 ps8661 ps8675 ps8691 ps8724 ps8855 ps8880 ps8928 ps9040 ps9140 ps9585 ps10197 ps10366 ps10633 ps10870 ps11118 ps11614 ps9073 ps9091 ps9472 ps9681 ps9913 ps9932 ps10202 ps10368 ps10634 ps10871 ps11121 ps9054 ps9075 ps9093 ps9474 ps9687 ps9914 ps9938 ps10203 ps10375 ps10636 ps10872 ps11178 ps9059 ps9077 ps9096 ps9476 ps9702 ps9915 ps9963 ps10206 ps10385 ps10643 ps10886 ps11300 ps9061 ps9081 ps9437 ps9478 ps9710 ps9917 ps9991 ps10218 ps10386 ps10672 ps10894 ps11304 ps9063 ps9083 ps9441 ps9480 ps9738 ps9920 ps9999 ps10278 ps10387 ps10686 ps10924 ps11336 ps9065 ps9085 ps9445 ps9482 ps9741 ps9923ps10280 ps10388 ps10745 ps11066 ps11588 ps9067 ps9087 ps9446 ps9488 ps9744 ps9925 ps10335 ps10408 ps10869 ps11103 ps11611 ps9069 ps9089 ps9447 ps9490 ps9757 ps9927

Something tripped the circuit that these machines were on and the breaker had to be reset despite not being in any way overloaded. They’re back up now and we have a datacenter technician looking into the cause of it.


12:10 pm

Web panel down

Posted (June 19th, 2009 at 12:10 pm PST) by andrea

The web panel is experiencing loading problems right now. Our admins are working on getting it usable again. Please check back here for updates.

We apologize for all inconvenience this is causing.

UPDATE 12:30pm PST: We’re back, web panel is working again, thanks for your patience!


11:20 am

Personal backup server problems

Posted (June 19th, 2009 at 11:20 am PST) by andrea

The backup server is up and operational for most of our customers, but some of you may experience connection issues or unable to see or access data previously uploaded. We are in the process of replacing some hardware on one of the server’s volumes. Please check back here for updates.

We apologize for all inconvenience this is causing.

UPDATE 1:20pm PST: Data rebuilding on above mentioned disk volume is in progress. No data loss is expected, however, due to the large amount of data on it will take a long time to complete. Our current, very rough estimate is early next week. Unfortunately, that also means that access and uploading new data to that volume is suspended until then. Again, very sorry about the inconvenience. Please continue checking back for updates.

Please also note: your user may have been temporarily affected by the problem this morning, even though it’s not actually on the bad volume. Please check as most of you have data access again.

UPDATE 2009-06-22 4:30PM PST: Our admins are still diligently working towards repairing the faulty drive. The volume rebuild has not completed. We thought that it would be finished by the end of the day. Realistically the process will be completed by tomorrow. We sincerely apologize for the inconvenience this has caused and appreciate your continued patience. Please check back for further updates.

UPDATE 2009-06-23 3:03PM PST: There was a hiccup on the fileserver that the backup machine is mounted to. This caused the drive rebuild to come to a halt. The rebuild is chugging along again. However, its only about 35% complete. Unfortunately its going to be another day or so before the process finishes. Again we apologize for the inconvenience and appreciate your patience. Please check back for future updates.


11:02 am

Web server falco is having problems

Posted (June 19th, 2009 at 11:02 am PST) by andrea

The machine has been having I/O issues and it looks like it will need a new raid card. A technician is on his way to the datacenter to replace it. Check this post for updates.

We apologize for all the inconvenience this downtime is causing.

UPDATE: 6/19/2009 12:15pm (Pacific): Falco is back UP and running again. Please check your sites and let us know if you run into any problems.


12:41 am

Dedicated hosting stability issues

Posted (June 18th, 2009 at 12:41 am PST) by patrickm

A routine backup script is unfortunately generating high loads on some of our Dedicated Customer Servers, which is causing some of them to crash.  We sincerely apologize for this and we are actively addressing the issue, so we hope to have the stability issues worked out as soon as possible.

Servers possibly affected are:

aeroquest1 artdesign astro bacon bandr baycouplesdb bcimages bcmweb bcreports bean bender bensons billbiz blazer bloodhoundgang bmw boney bootifull brookeburke bscmanagement btia c100webserver cadet calgary calmini carlotta catering cbjdigital1 cgbny charleston clickmeister cobb cobblers codlib cornerhw cowboy creek dagon dante deimos destro devtvr dicta dissenter dixie dsl eightoneeight embassy esdraelon evangeline evolution01 factorycat falcor fepi foreplay freeco futuretrax gauche gazoline gendou gmlb1 gnu greysoft1 greyson groundcontrol hoodcore hutllan hypnopics idpfilm ifpserver ifthen inaux infoword jbeck kidalex killeroldies kittens kittenwar kmi kylie labcomp lifetimesvoice lincoln loki lowest-airfare luke matisse max mbserver meep motmot mqseries mushroomhead naish nassweb neversoft nwtekno obraxus odt oshea p5audio padrino pappocom pariso perfecto peterpan phobos picasso ping polaris principa prometheus reallifelab rlharvester saints sauron sax scudettoweb searchmeister secrets settlements shazam silversun singe sitepartner suba sunimage sunmachine swingdater tagrel tekhost telkoth thetinagrant timber toddmuchmore trackseven traveltracer utopia vantec vmachine wbcentral webwerks wireandtransit wlusp womble wpcf yaleweb zartan

We will update this post with more information as soon as we can!

UPDATE 1:32AM (Pacific): All services should be restored now, but please contact Support if you are still experiencing any downtime, or stability issues!


3:54 pm

Saltine Moving to New Hardware

Posted (June 16th, 2009 at 3:54 pm PST) by Leon

The web server, saltine, is being moved to new hardware as it is failing come back up from a reboot. Saltine should be back up within the next 30-45 minutes… Check back for more updates!

Update 4:53PM: Sorry that it took longer than expected but saltine is back up and happy!

– The Happy DreamHost Server Failover Robot


10:16 pm

Falco is fsck’ing

Posted (June 15th, 2009 at 10:16 pm PST) by Sandon

The web-server Falco is currently down as errors were detected on its file-system bringing it into read-only mode. An fsck (file system check) is currently running on it to get it back to normal status. This typically takes around an hour but could be up to two.

Please check here for updates.

Update 11:40 PM PST: The fsck is still running strong. no ETA as there is really no way to know for sure how long a fsck will take. It is very rare for it to take over 2 hours though.

And I apologize for typo’ing the server name as ‘faclo’ as it is indeed ‘falco’ as one of the commenter’s mentioned.

Update 01:29 AM PST: Unfortunately this is taking longer than expected. Its on pass 1B which is rescanning for multiply-claimed blocks. I will update this again as soon as its completed.

Update 2:55 AM PST: Sorry for the delay guys! It looks like the fsck finally finished and the machine is comming back up now and so far appears to be good now. It looks like the fsck might have taken a while due to the large number of files on the machine. This machine appears to have 33 million inodes being used which is quite a bit more than our average server.


3:04 pm

favicon.ico, favicon.gif and robots.txt update

Posted (June 10th, 2009 at 3:04 pm PST) by kitchen

In the interests of server stability, we have proactively started creating favicon.ico, favicon.gif, and robots.txt files in domain directories. We are not overwriting any of these files, only creating them should they not exist.

The reason we are doing this is due to software packages using mod_rewrite rules to redirect requests for non-existent files to a default handler (such as a php script, a rails application, etc), which is an acceptable and perfectly valid use of the mod_rewrite rules, however when such a site is missing files, and starts receiving a heavy load of traffic, it can severely adversely affect server performance and stability.

Most browsers request the favicon.ico file by default when making a request, and additionally some even ‘pre-load’ the favicon.ico file from linked sites when visiting other sites. Some browsers even look for a favicon.gif file, though this usage appears to have dropped off considerably, we decided we should probably go ahead and create these as well, just in case.

Additionally, well-behaved web crawlers will request the robots.txt file to see what they should or should not be indexing at the site operator’s request. While under normal circumstances they generally only request the file once, prior to indexing, we’ve seen behavior where a web crawler will repeatedly request the file, in itself causing server instability and load issues.

To attempt to alleviate these problems, we’ve started creating the favicon.ico, favicon.gif, and robots.txt files in all domains across our system. The files are blank, 0 byte files, and simply serve to prevent things like WordPress from causing extremely high loads and server instability in the event of a traffic surge to the site, simply because it’s doing a whole lot of unnecessary work to generate a 404 page, which itself isn’t useful to the browser either.

You can easily replace these files with those of your own creation simply by uploading a new one to the server. We only create them where they do not previously exist. If for some *crazy* reason, it decides to ‘create’ the files again anyways, we simply use the ‘touch’ command, which on an existing file will simply update the last modified timestamp, not affecting any of the contents of the file whatsoever.

Please keep in mind that the creation of these blank files is entirely harmless, and will only serve to speed things up on your site and system-wide. It is in *your* best interests that we create these files.

If you have any questions, please file a support ticket and we’ll be happy to address any of your concerns.

Sincerely,
The Happy DreamHost Server Stability Robot


 
© 1996-2009, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
2:49 pm

Ram Upgrades!

Posted (June 9th, 2009 at 2:49 pm PST) by micah

We will be performing quick RAM upgrades on some of our more loaded servers today at 4pm. Each server should only be down for 5 minutes or so. We will do everything possible to minimize the downtime and hopefully they should all come up performing a lot better! Only HTTP, SSH, and FTP should be affected by this maintenance.

The list of servers is:

greene
hulk
schultz
bonanno
castiglia
burke
fay
watanabe
gambino
barker

Please contact support if you have any questions.

Happy DreamHost Special-Talent Upgrade Team!

Update — These are all finished as of Tue Jun 9 17:30:56 PDT 2009


12:25 pm

Moving Persephone to New Hardware

Posted (June 9th, 2009 at 12:25 pm PST) by Kyle

On numerous occasions persephone has had problems related to it’s network interface card and we think that it’s high time to get everything over to a new piece of hardware before it tanks in the middle of the night.  Estimated downtime is 20-30 minutes.

UPDATE: The migration to new hardware is complete, if you are still experiencing issues please contact support via the panel.


11:53 am

Server “badillo” is having a bad day!

Posted (June 8th, 2009 at 11:53 am PST) by David

It looks like server “badillo” is having a bad day and needs new hardware. One of our admins is on the case and is moving all accounts on that server to new hardware as we speak. The whole process shouldn’t take too long. We’ll make another post once the process has been completed.

We apologize for any inconvenience this may cause.

UPDATE: 12:05 PDT - Unfortunately because there was also a problem with the spare server that was to be used another new server is being deployed as a replacement. This means the whole process could take about 1.5 hours to complete. We apologize for the longer than normal time it will take to restore services for your server.

UPDATE 1:01 P.M. PDT – Badillo is back up and running. Sorry for the inconvenience. If you have any further problems with this server, please contact support!


10:16 am

Emergency Network Outage

Posted (June 8th, 2009 at 10:16 am PST) by Patrick

One of our network links just went down due to failed hardware. We have replacement hardware on hand and are installing it now. During this installation much of our network will be inaccessible. This downtime should not exceed 10 minutes.

Check this space for updates.

— Update Mon Jun 8 11:09:52 PDT 2009 —

The hardware has been replaced and the network has been up and running properly for the past 30 minutes or so. There was some fallout due to the network being down, but that has also been cleaned up at this time.

If you are still having problems connecting to your sites, please contact support.


6:49 am

Spunky cluster outage

Posted (June 8th, 2009 at 6:49 am PST) by patrickm

We are currently experiencing some downtime with websites hosted on the Spunky cluster, due to a downed file server.  We are currently working on getting this back up and running and we apologize for the inconvenience that this is causing.  Please check back here for updates, as soon as they become available!

Servers that may be affected by this are:

absinthe achilles adams ak alondra altoid atlantic aviation babyruth badillo bandini barranca baseline bazooka beaudry bellflower betsybird beverly blowpop bonbon bottlecap bounty bourbon boyd broadway brownie bundy butler butterfinger cahuenga centinela cerritos cheese cheeto cheezit chiclet cienega cognac colbert cookie crackerjack cupcake dingdong dorito drumstick dumdum fairfax federal fireballs firestone fletcher flower formosa frito fruitland fudge fundip garth garvey good gower grand granola grappa gumdrop gummi hillhurst hillmont hoho holt hope hyperion jalapa jefferson juniormint jurupa larchmont laurel lemonhead licorice lifesaver limbo-spunky4 look lucas mariposa mayo mazalien melrose milliken mills monroe moorpark nash necco nightlife nilla nothingless nougat nuno nutrageous oakdale obvious ocean ord overland oxnard paramount payday pernod pez piggy plenty popcorn pretzel ps4769 ps5117 ps6496 ps6519 rampart rangerxxx razzle redhot redondo reeses rexford ritz rockyroad rodeo rollo roscoe rosecrans rum saltine sanborn schnapps scotch selma sepulveda shoutitlouder skittle skor skybar smores snicker soto sourpatch splashy spree stravinsky suburbview sugardaddy sunset syt taffy tampa tequila toffee tootsie truffle twinkie twix twizzler union vanowen vermouth vine virgil washington wayne western whittier whopper wilmington wilshire wiredgeek wmo york

UPDATE 6:57am Pacific: We are also adding Spunky email to the list of services that are down due to this, as some email accounts are hosted on that file server.

We have also discovered the cause for this downtime.  The shelves that hold this file server are redundantly powered, but it seems that both of the power supplies have taken a dive.  We are currently working to restore power to this shelf and hope to have that completed soon.  Check back here for more updates!

UPDATE 7:33am Pacific: We’re almost there!  We ended up replacing the shelf for this file server, as we discovered that one of the power strips died and the power supply connected to the hot strip was also dead!  So, this normally wouldn’t have caused an outage like this, save for the surprise dead power strip.  The file server is coming back online now and we’ll post the all clear here when we have completed some tests.

UPDATE 10:45am Pacific: The file server issues have been resolved and all services appear to be working correctly now. We are still however dealing with the newest status post about network issues we are dealing with.

http://www.dreamhoststatus.com/2009/06/08/emergency-network-outage/

So if you are noticing problems still it is likely related to that outage now. Sorry about the file server downtime. Typically our backup power supply would have saved us from this outage. This issue is resolved.


12:57 am

Exodus is down for the count.

Posted (June 6th, 2009 at 12:57 am PST) by Sandon

It looks like one of our web-server ‘exodus’ was unresponsive and had not came back up after we attempted to power-cycle it remotely. Right now an admin is en-route to the data center to check out the situation.

Please check here for updates on this server’s status.

Update: 02:19 AM PST. it looks like the problem was with the raid controllers BBU that was preventing it from booting. It has been replaced and should be back up shortly.

Update 02:30 AM PST. Ok the server and all its services should be restored.


6:14 am

Email Authentication issues

Posted (June 5th, 2009 at 6:14 am PST) by patrickm

We are currently experiencing an issue with email authentication, which is causing an outage of email services.  We are currently addressing this issue and hope to have it resolved shortly.  Please check back here for updates, as they become available.  We sincerely apologize for the inconvenience that this is causing.

update: This issue appears to be resolved and only lasted for about 10-15 minutes.  We found that the ‘imapproxy’ service stopped running on a number of mail servers, as well as some issues with the SMTP service.  We re-initialized these services and they came back quickly.  If you are still having any issues, please feel free to contact Support and they will be happy to check!


10:15 am

Tick down for maintenance

Posted (June 4th, 2009 at 10:15 am PST) by andrea

The web server ‘Tick’ is going down for maintenance. It should take less than an hour, but check back here for updates. Sorry about the inconvenience this may be causing.

Update 11:27am: Unfortunately, it turns out we need to migrate customer data to a more stable server, as even though Tick came back up and working, the same issues have resurfaced. Data is being transfered, and shouldn’t take longer than a day before all custromers on Tick are ready to go on the new machine. Tick is serving data for now, but let us know if you need assistance in the meantime. Again, sorry about all inconvenience this is causing.

Update 12:55pm: The data move has finished. If you used to be on Tick and now you are on Twofaced and have problems with your site or other services, please don’t hesitate to contact support with the details.


1:40 pm

fay server having problems.

Posted (June 1st, 2009 at 1:40 pm PST) by danwy

Our server, fay, has been having issues with its load today. Our admins are trying to track down this issue and get it running smoothly again, but our attempts this far have been unsuccessful. We’re not sure when this will be fully resolved, but we’re trying to get this running smoothly ASAP. We’re sorry for any inconvenience this causes you.

Update: 4:30pm PST – Things are looking better for this server now. The load has been at an acceptable level for the last hour or so. Our admin team will continue to keep an eye on this server though in the event that it starts acting up again. Please contact support if you have any further issues on this server.

Update: Jun 2 2009 10:00am PST – After a long period of stability the load has creeped back up on this server. Our admin team is investigating this again in hopes of finding the cause. We’ll keep this post updated with its status.


6:30 am

Homie cluster downtime

Posted (May 28th, 2009 at 6:30 am PST) by patrickm

We are currently experiencing downtime with our Homie cluster.  Our Admins are actively working on this issue and hope to have it resolved shortly.  We sincerely apologize for the inconvenience that this is causing.  Please check back here for updates!

UPDATE – 12:45pm PST: This problem should have been fixed for the last couple of hours, but we’ve been testing things out to make sure everything is back to normal. This issue was caused by a php upgrade to our homie server image that was done for security purposes and that caused the server apaches to become unresponsive until they were restarted. We scheduled the restarts as soon as we isolated the problem, but due to the amount of servers that needed this done it took a while to get them all updated. We’ve tested all the servers affected, along with a large pool of customer domains on these servers, and they seem to be working fine now. If you have any additional issues, it was likely not due to this incident and you should contact support for further assistance. We’re sorry for any inconvenience this caused you.


 
© 1996-2009, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
8:51 am

Homie Cluster Server Load Issues

Posted (May 21st, 2009 at 8:51 am PST) by mir

We have been experiencing issues on a number of servers in our Homie cluster where machines will perform well but then experience extremely high loads and even crash on a regular basis. It appears to be a different issue in each case so we are having to investigate each of them individually:

avalanche (believed to be resolved but still under careful watch)
capone (stability has improved after a few fixes)
cyclops
nightwing (a couple of abuse issues addressed, waiting to make sure that was it)
humantorch
zatara (greatly improved already but still being monitored)
giancana
terranova (load stable after heavy users have been removed but still being monitored closely)
batman (already greatly improved and finishing up with a couple of heavy users – update: one more heavy user found and removed, load stabilized)
nitti (offloaded users and looks a lot better)
pistone
galactus (we are offloading this server and have found a few TOS violators we are disabling)
thing (also offloading accounts and looking into heavy resource usage accounts)
dillinger (we have dealt with a few bad users and are offloading as well)
legs (load seems low and steady for now, possible apache web server issues)
luciano (load is low and steady, being watched)

Please rest assured that this is the highest priority for both the admin and tech support teams and that we are doing our best to get this resolved once and for all (we are moving customers off of most of these machines as we investigate as well as this will help alleviate the load issues no matter what the final determination in regards to cause turns out to be).


4:05 pm

DreamHost central services maintenance

Posted (May 18th, 2009 at 4:05 pm PST) by DreamHost

DreamHost central services will be taken offline for database maintenance starting Thursday, May 21 at 10:00PM PDT, extending for 1 hour until Thursday, May 21 at 11:00PM PDT. The following services will be offline during this time:

DreamHost Webpanel
DreamHost-provided webmail
DreamHostApps


1:57 pm

riffraff MySQL server problems

Posted (May 18th, 2009 at 1:57 pm PST) by danwy

We’re currently having some hardware issues with the riffraff MySQL server. One of our admins is currently heading down to the datacenter to recover it. We don’t have an ETA on when this will be resolved yet, but we’ll update this post as soon as we have more information.

Update – 3:57pm PST: This server has been back up and running for the last 30 minutes or so and we’ve verified that things are now running smoothly. If you’re still having problems it was not likely related to this server and you should contact support to investigate it further. We’re sorry for any inconvenience this caused you.


8:09 am

changes made in Web Panel delayed

Posted (May 18th, 2009 at 8:09 am PST) by patrickm

Changes made within the Web Panel are being delayed at this time.  This includes just about everything (OneClick Installs, Database creation/changes, User creation/changes, Password updates, etc.) .  The queue for these requests is backed up at this time, but is processing the requests, so please allow more time for these changes to happen in the system.

We hope to have this issue resolved as soon as possible, but please feel free to contact support if you’re unsure if this issue is affecting you.

UPDATE 1:10pm PST: This is now resolved. All updaters are again running as scheduled, and all the ones scheduled prior to this have caught up. Thanks for your patience!


7:19 am

broken axl

Posted (May 18th, 2009 at 7:19 am PST) by patrickm

The network card in the “axl” server died on us and we’re in the process of replacing the hardware for that server.  This process should only take about 30 minutes, but we will post here with any updates!  Thanks for  your patience and we apologize for the inconvenience that this has caused!

Update: Sorry for the late update, the server was back to work about 45 minutes after the inital post, if you are still having problem with your site and your account is on axl please contact support.


3:48 pm

Backup server issues

Posted (May 17th, 2009 at 3:48 pm PST) by danwy

We’re having some issues with the customer backup server (for storing backups/archives). One of our admins is going to have to head out to the datacenter to manually work on the server. We don’t have an ETA on when this will be resolved, but it’s something that our admins are aware of and should be back in business soon. If you’ve written into support about this and haven’t received a reply yet, we will be replying to them once the issue has been resolved.


6:49 pm

ProfessorX is having a bit a trouble!

Posted (May 15th, 2009 at 6:49 pm PST) by Brian@dreamhost

Unfortunately, it looks like our server “Professorx” has crashed. We’re currently working on getting it backup, and online. This is taking a little longer than expected, as we have to check the disks on the server, as it comes up. We should have it up and working soon, and apologize for the problems related to this!

— Update Sun May 17 14:28:28 PDT 2009 —

The professorx server was up and running normally for a day or so but the disks are showing further issues. We have had to run a disk check again to get them in working order. Unfortunately they cannot be accessed during this time so there will be some downtime until the check has completed. We will update this post when we have further info, or if the server will need to be moved to new hardware. Our apologies again for the issues with this server.

— Update Sun May 17 15:08:23 PDT 2009 —

The server is back up and running now, as should all sites and FTP abilities. The investigation will continue and we’ll decide shortly if this server needs to be moved over to new hardware. We’re sorry for any inconvenience this caused you and we are doing all that we can to prevent this from happening again in the future.


10:18 am

problems with bullseye

Posted (May 14th, 2009 at 10:18 am PST) by danwy

We’re experiencing some issues with the raid setup on bullseye. Our admins are working on this issue now and should have it fixed up shortly. We’ll update this post as soon as it is resolved.

UPDATE – 11:29am PST: This should be resolved now. Our account mover script was a little overzealous and tried moving a whole chunk of accounts to it all at the same time instead of one at a time. We’ve killed these moves and things should be running smoothly now. We’re sorry for any inconvenience this caused you.


12:49 pm

UPCOMING: Shared HTTP Server Outbound Mail Policy Change

Posted (May 12th, 2009 at 12:49 pm PST) by glen

Starting May 19, 2009 we will start rolling out to our shared HTTP servers a block on all outgoing mail attempts that establish a direct MX connection over port 25 . A gradual cluster by cluster roll-out will ensue to ensure that there are no unexpected widespread issues. This is being done as we’ve observed that only a minuscule percentage of our users use this for legitimate mail, while a large percentage of spam complaints come from this method that bypasses our sendmail and sender quota setup. This will have ZERO affect on on mail sending from e-mail clients and via webmail. This would only potentially affect mail originating from websites on shared HTTP servers that do not use our standard mail transfer setup.

What Does this mean to my website?
Likely Nothing! 99.8%+ of our customers websites use our sendmail Mail Transfer Agent (MTA) which will continue functioning as is . We are simply blocking connections for any software that uses its own MTA . If your site is using the mail() function via PHP, or any similar scripting language, you will not be affected. If you are using /usr/sbin/sendmail , you will not be affected.

So You are Blocking Port 25?
Sorta! We are blocking OUTBOUND access to port 25 by all scripts that are not our own MTA and are not connecting within our Network . Using an SMTP connection through an e-mail app or script is considered an INBOUND connection. As well if you are using AUTH SMTP to our Mail servers the connection will still go through. Nothing will change in that regard.

Why are you doing this?
Spam! Spam! SPAM! All too large a percentage of spam complaints and blocks we are seeing lately are due to spammers running their own MTAs from hacked accounts. These MTAs circumvent our existing anti-spam measures , which lead to the unfortunate result of blacklisting that affect the flow of HTTP server based mail for all customers that share the server.

I have legitimate reasons for running my own MTA or connecting to port 25 outbound!
We’d encourage you to start using our default sendmail setup. Otherwise we will not be adding this block to Private Servers, so upgrading is always an option if you are going to want to run your own mail service . If you have any questions about your use contact support.


Will my Contact Form, WordPress, Drupal, PhpBB, ZenCart or other One-Click install be affected?

No. These and most common scripts use the mail() function using our default MTA . They will not be affected.


11:28 am

Homie cluster mail authentication issues (again)

Posted (May 11th, 2009 at 11:28 am PST) by danwy

We are having problems with authentication on our homie mail servers again for a small portion of our customers. Our admins are currently investigating the issue and hope to have service restored again shortly. We’re not sure if it is the same problem as last time (which we thought we resolved), but we will post more information here as soon as we have it. We apologize for any inconvenience this causes you.

Update: May 11, 12:30pm: This issue should be resolved now. Unfortunately, we’re not quite sure what the cause was. We rebooted the mail servers to get a clean slate and they came back up running fine. We’ve been testing mailboxes that were having issues according to our customers and weren’t able to duplicate any errors. If you have further issues with this, please contact support, as it is likely something unique to your specific account. Again, we apologize for any inconvenience this caused you.


 
© 1996-2009, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
5:29 pm

Bowser was defeated

Posted (May 7th, 2009 at 5:29 pm PST) by Leon

The web server, bowser, keeps having issues with its NIC so we’re moving it off to new hardware. It should be back up within the next 30 minutes…

Update 6:48PM PDT: Sorry that it took longer than expected but the server should be back up! If you’re still experiencing issues, please contact support!

The Happy DreamHost Mario Playing Team!


3:58 pm

Mona server having issues

Posted (May 6th, 2009 at 3:58 pm PST) by danwy

We are currently looking into some issues on the mona server that is causing the load to spike. It initially looked like it may have been network related, but the more we investigate it the less likely that seems. Our admin will continue to investigate the problems on this server and hope to have it resolved shortly. We’re sorry for any inconvenience this causes you.


7:53 am

Homie cluster mail authentication issues.

Posted (May 6th, 2009 at 7:53 am PST) by Sandon

It looks like one of our clusters (homie) is having a problem with mail authentication which is causing users to be unable to access or send email. Our admins are currently working on resolving the problem.

You can check which cluster you are on from the panel by clicking “Account status” in the upper right in the “Your Email Server” field. Please check here for updates.

— UPDATE Wed May 6 09:14:35 PDT 2009 —

The admins have corrected the problem and email service should be restored. We are running some more tests to verify that all is working normally again. We identified a problem where mail servers were not using all available authentification databases, they are now correctly setup which will improve reliability of email due to more redundancy. Our apologies for the problems this morning with email.


6:47 pm

Planned mail downtime – Homie mail server – Sunday May 3

Posted (May 1st, 2009 at 6:47 pm PST) by Ralph

This planned downtime is actually for a small subset of users who have Homie listed as your email server. You can see if you are on homie from the DreamHost web panel when you click on Account Status in the right corner of the panel. Unfortunately at this time we do not have any clear way to see if you are in this group of users so please check the planned downtime time frame if you experience mail issues this coming Sunday May 3rd.

The planned downtime will start at 6pm pacific time on Sunday May 3rd and will last for approximately 3 hours. We will be upgrading mail service in this window. Mail service will be down, but messages sent to you will typically be deferred for later delivery. If the sending mail server does not defer messages than there will be a bounce, but most servers have a deferred system in place where it will try to deliver the message again at a later time. Once the maintenance is started we will update this status post followed up by another post once the upgrade is complete.

UPDATE 8:54PM **
The maintenance has been completed around 45 minutes ago. All homie mail users should be working as normal. If you are still experiencing any kind of problem please notify support.


1:49 pm

batman/dio/deathlok/johnnyblaze having some problems

Posted (April 30th, 2009 at 1:49 pm PST) by kitchen

We’re currently having some issues with the batman server and our admin team is investigating the situation right now. We apologize for any inconvenience!

Update – 3:16pm PST: This also affected a few other servers as well – dio, deathlok and egon. The batman, deathlok and egon servers are on their way back up now, but we’re still working on dio. We hope to have that back up and running as soon as we can as well. We’ll resolve this post once we’re sure these servers are running smoothly again.

Update – 10:30pm PST: We’re still working on getting these machines back up and restored to their former glory. Egon should be completely restored now. Batman, deathlok and dio are back up and working, however, some data is still not there. We’re copying over that from backups now, so if you’re still missing data, hold tight, we should have that restored soon. Johnnyblaze seems to be the hardest hit, so we’re working feverishly to get this back in working order.

Update – 8:15pm PST May 1: Our recovery efforts are now complete and to those of you who have lost data we sincerely apologize and you should have a message in your inbox detailing any compensation due to lost data. If you are still having problems or if you did not receive direct contact from us about data you have lost please contact support immediately.


12:12 pm

Luigi being DDoS’d

Posted (April 30th, 2009 at 12:12 pm PST) by danwy

Our server, luigi, is currently being hit with a DDoS attack. If your site(s) hosted on this server and is experiencing issues, this is why. Our admins are working on this now and are doing all they can to minimize the effects of this attack and get things back to normal. Please check back here for updates, we’ll post new information as we have it.

UPDATE (May 1 – 4:00am PST): We feel confident that this issue is resolved, as the traffic caused by the DDoS to this server has been blocked! The load on the server is great and sites are able to resolve once more. Of course, if you continue to have any issues, please contact Support and they’ll be happy to check it out!


2:14 pm

Panel downtime – RESOLVED

Posted (April 29th, 2009 at 2:14 pm PST) by patrickm

We are currently experiencing an issue with the Web Panel which is causing it to be inaccessible.  We sincerely apologize for this downtime!  We are working to resolve this issue as quickly as possible.

Please check back here for more information, as it becomes available.

UPDATE: Again, sorry for the lack of updates on this particular issue. This was actually resolved shortly after this was posted, but we wanted to be absolutely sure that the Panel was working reliably before resolving this post. As always, if you have any issues with Web Panel access, please contact Support and they’ll be happy to help!


12:24 pm

Quid needs a boost

Posted (April 28th, 2009 at 12:24 pm PST) by Terri

Quid’s boot disk went bye-bye, so I’m moving everyone over to new hardware. Very sorry about the inconvenience, but everyone should be back up and running within 30 minutes to an hour.

Thank you all for your patience!


7:51 am

Junk Filtered Email Delays

Posted (April 28th, 2009 at 7:51 am PST) by Justin@DH

We are investigating some delays in delivery of email, specifically for junk mail filtered email addresses. Mail is being processed now and we expect it to be caught up soon, but we are still collecting information. When we have more info or an ETA for a fix we will post an update.

UPDATE: We apologize for the lack of updates on this issue. We have been closely monitoring the junk filter queues to make sure there are no remaining issues. We feel strongly that this issue is behind us, but if you continue to see any delays, please feel free to contact support!


8:39 pm

Felix went splat

Posted (April 27th, 2009 at 8:39 pm PST) by Jason C

Users on the MySQL server “felix” will not be able to connect to their databases due to some unforeseen downtime. One of our admins has been dispatched to work on the problem and hopefully there will be a resolution soon. More updates will be posted here as things develop.

Update – 9:39 PM PST: It’s looking quite likely that we’re going to have to restore this server from backups. We’re still working towards a solution tho — so don’t think we’ve forgotten about you.

Update – 2:37 AM PST: Felix is back online. All databases were restored from the morning backup.


 
© 1996-2009, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
4:54 pm

Changing the name of backup.dreamhost.com

Posted (April 27th, 2009 at 4:54 pm PST) by kitchen

Since we have customer-accessible backups at backup.dreamhost.com and non-customer accessible backup servers, having the customer-accessible one named ‘backup’ is getting very confusing around the office. Because of this, we are renaming the server to ‘hanjin.dreamhost.com’.

How will this affect you? It shouldn’t, at all. The machine will still be accessible via ‘backup.dreamhost.com’, but any ftp/ssh banners will refer to ‘hanjin.dreamhost.com’. We will not be deprecating the use of ‘backup.dreamhost.com’ any time, this is mostly just an internal change so we can keep our sanity when talking about backups vs backup :)

The Happy DreamHost Backups and Maintaining Sysadmin Sanity Team


11:54 am

Castellano is moving to new hardware

Posted (April 27th, 2009 at 11:54 am PST) by andrea

Two disks of the current machine’s array died and are being replaced, but the rebuilding is overloading the machine (you probably noticed the extraordinarily high loads just a few minutes ago), and thus making it pretty much impossible to provide hosting services on it. We decided to move it to new hardware instead, and starting to copy over the data right now. We try and get the new machine up as Castellano, and apologize for the inconvenience this load and transition is causing. We’ll update the post as soon as we make progress.

UPDATE 1:10pm PST: We’re continuing to move the data to the new server from backups. As this is a very long process, we turned back on the old machine for the time being. The load is a tad higher than preferred, but the machine is operational. Once the backup data is transferred, we’ll sync the two machines and turn the new one on. We appreciate your patience during the process. We’ll update this status post if there is any change or when the transfer is finished.


9:18 pm

DreamHost Central Services Maintenance – Happening Now!

Posted (April 21st, 2009 at 9:18 pm PST) by DreamHost

UPDATE: 12:56am Wednesday 4/22, All services are back online.

As previously announced here (Post of April 14, 2009 re. Central Maintenance), DreamHost central services have been taken offline for database maintenance. We started “on time” at 10:00PM PDT, Tuesday, April 21 , 2009, and expect the maintenance window to extend approximately 4 hours until Wednesday, April 22 at 2:00AM PDT. The following services will be offline during this time:

DreamHost Webpanel
DreamHost-provided webmail
DreamHostApps

We apologize for any inconvenience this may cause, and appreciate your patience while we work to improve your hosting experience.


4:39 pm

DreamHost central services maintenance, next Tuesday

Posted (April 21st, 2009 at 4:39 pm PST) by DreamHost

DreamHost central services will be taken offline for database maintenance starting Tuesday, April 21 at 10:00PM PDT, extending for 4 hours until Wednesday, April 22 at 2:00AM PDT. The following services will be offline during this time:

DreamHost Webpanel
DreamHost-provided webmail
DreamHostApps


3:17 pm

Homie cluster users seeing some problems

Posted (April 20th, 2009 at 3:17 pm PST) by Jason C

Due to some issues with one of the central servers that all the machines in the homie cluster use, users on those machines might be seeing some outages currently. Our admin team has scrambled to fix this tho and hopefully we’ll have a resolution for you right quick. Keep an eye on this post for updates.

The affected machines are:

adonis agamemnon ajax altoid angel aphrodite apocalypse apollo aquaman aries atlas avalanche baachus banshee barefoottours batman bishop blade brasco bugsy bulger bullseye cable calypso cannonball capone catwoman charon charybdis chronos cloak colossus coolblue coonan cummiskey cyclops daedalus dagger deathlok dillinger dione dionysus eastman electra erebus erida eros exodus fitzpatrick floyd galactus gambit ganesha ghostrider giancana gorgon hades havok hawkeye hawkman hecate hector helios hephaestus hera heracles hermes hestia hobgoblin homie-vserver32 hulk humantorch hypnos icarus iceman iris johnnyblaze joker jove jubilee kakuji kingpin kratos larchmont legs leto magneto masseria mclean medusa midas minerva morpheus mothman mrfantastic mystique namor nemesis netboot-testy nightwing odysseus oedipus osiris pan pandora pegasus penguin persephone pistone poseidon professorx proteus quicksilver rhea riddler sabretooth sandman silversurfer sisyphus snocap snowbird spectre spiderman spillane storm tartarus terranova tethys thanatos thetis thing triton tyche venom vulcan walsh wasp wilshire wolverine wonderwoman zagnut zatara zeus zwerbach

And DreamhostPS machines:

ps7051    ps7093    ps7146    ps7209    ps7403    ps7477    ps7736    ps7763    ps7866    ps8160    ps8202    ps8741 ps7073    ps7095    ps7177    ps7345    ps7424    ps7662    ps7737    ps7779    ps8035    ps8161    ps8220 ps7080    ps7096    ps7183    ps7355    ps7429    ps7734    ps7738    ps7847    ps8056    ps8162    ps8252 ps7087    ps7097    ps7208    ps7356    ps7473    ps7735    ps7739    ps7865    ps8159    ps8167    ps8254

Update: The bad hardware was replaced and the machines appear to be recovering now. If you’re still having issues, please contact our support staff!


10:42 am

backup.dreamhost.com short downtime

Posted (April 20th, 2009 at 10:42 am PST) by kitchen

At 1PM Pacific, we will be taking backup.dreamhost.com down for a few minutes to move it to another rack in our datacenter. The downtime should be no more than 30 minutes, and we will post an update once things are back up and running!

Update: 2PM Pacific: We’re taking backup.dreamhost.com down now, it should be back up and running within about 30 minutes!

Update: 2:45PM Pacific: backup.dreamhost.com is back online! Thank you for your patience.

The Happy DreamHost Backup Machine


7:15 am

Spam Filter Email Issues This Morning

Posted (April 20th, 2009 at 7:15 am PST) by Justin@DH

We are experiencing issues with email addresses that are using our spam filtering service. The errors are 554 or relay denied when sending from outside email addresses. We are working to get this issue resolved as soon as possible, and admins are working on the issue right now. Our apologies for the problems. When we have further info we will post it here.

Update 1:30pm PSTAccording to the admin working on this, most email addresses should be able to receive outside mail again without errors. We’re double checking right now and running mass configs on the mail servers to make sure we didn’t miss any. Those should finish by the end of the day. If you’re still having issues tomorrow morning, please do let us know, so we can look into it for you. Please provide details, so we can better assist you. We’ll update this post as soon as all email addresses are reconfigured.

Update 5:40pm PST We believe that all incoming mail is now operational; all updates have been completed. If you’re still having problems with mail, please submit a ticket through the web panel with as many details as possible, so we can look into it for you. Sorry again for the inconvenience this has caused earlier.

Update 8:37pm PST There were some users who may have been missed when the all clear came down the pipe a few hours ago. Just to be safe, our admin team is running the cleanup scripts again. This will get anyone still seeing bounces (of which there should only be a handful at the most) cleared up in a few more hours.

Update 9:54pm PST I’ve been given a proper “all clear” by the admin in charge of this cleanup process. If you’re still seeing issues, please contact technical support.


2:00 am

Private Server Host Outage

Posted (April 18th, 2009 at 2:00 am PST) by Kyle

One of the Host servers for our Private Server customers is having disk array problems so we are migrating over to new hardware. Affected guests:

ps8373 ps8375 ps8376 ps8377 ps8378 ps8379 ps8381 ps8382 ps8385 ps8390 ps8394 ps8395 ps8396 ps8397 ps8398 ps8399 ps8400 ps8401 ps8402 ps8404 ps8406 ps8758 ps8772 ps8779 ps8782 ps8784 ps8786 ps8787 ps8789 ps8790 ps8791 ps8792 ps8793 ps8794 ps8795 ps8796 ps8797 ps8798 ps8980 ps8982 ps8984 ps8986 ps8988 ps8990 ps8992 ps8995 ps9003 ps9004 ps9134 ps9137 ps9141 ps9145 ps9150 ps9162 ps9493 ps9494 ps9495 ps9499 ps9500 ps9577 ps9581 ps9586 ps9592

Update 8:20 PM PDT: Sorry for the lack of updates and the time it took to bring this back up! In any case, we have finished the migration for the private server and all of your servers should be back up! If you’re still experiencing issues, please contact our support team!


8:33 pm

Bulger Not Feeling So Hot

Posted (April 17th, 2009 at 8:33 pm PST) by Leon

The web server, bulger, is having issues with its raid and attempts to move the drives over to a new server have also failed. We’re currently bringing up a new server and restoring it from backups… More information to come!

Update: Further hardware disaster with this machine has extended downtime, unfortunately. We are still working on it and we hope to have it back up soon. We are very sorry for the inconvenience.

Update 06:35 PDT: The machine is back up now but is still restoring from backups and some configurations are still running on the server. Websites should be up for the most part but if they’re not, don’t fret as we’re still working on it! We’ll post more updates as they come in!

Update 8:11 PM PDT: The server is still restoring from backups but sites should be online. You will notice a bit of instability meanwhile all the backups are restored but this is only temporary and subside once everything is restored. We’ll post the all clear once everything is good to go!

Update 4/20/2009 12:30pm PSTJust a quick update to let you know that the restore is still going. All sites are be operational while data is moved to the new machine. You can upload new data to the server as you normally would. The only thing that you may be noticing is some slowness due to the transfer of backup data to the new server’s local drives. This is a very long process, and we apologize for all inconvenience this is causing. We’ll update this post again when the data restore has finished.


6:38 pm

Backup Service Update

Posted (April 16th, 2009 at 6:38 pm PST) by Ralph

The missing backups are currently being restored. We are restoring them with an option that will not overwrite any new backups you have created. The backup service itself is still having some stability issues. Our plan of action on the service will begin on Monday. This will involve restoring the backups so they can be accessed from your current backup user, adding more disk space to the service, and improving stability of the service. This maintenance on the backup service will begin on April 20th but it will be a process that requires multiple steps to complete. Sorry the service has been unreliable, we are definitely working on making it a better offering for you.

Update April 20th 18:45: Some of you have already seen our post today detailing a move of the backup service:

http://www.dreamhoststatus.com/2009/04/20/backupdreamhostcom-short-downtime/

We are expecting to almost have the backups restored by tomorrow at some point. I will update any further details at that time once we know if our guesstimate is accurate or not. Issues with the backup service are definitely nearing their end.

Update April 21 12:30: The backup service is almost exactly where we want it to be. We just have a rsync running making sure we have all the data properly merged. This is just one part of the backup service. We feel the service is stable now and besides some minor shuffling we are pretty much out of the woods. If you find you are missing files that you may have uploaded after midnight our time last night, don’t worry, those are in the final rsync we are running.


 
© 1996-2009, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
7:47 pm

Fundip Not So Fun!

Posted (April 15th, 2009 at 7:47 pm PST) by micah

Fundip went down a couple times today with errors that lead us to believe that it’s hardware is failing. We’re in the process of failing it over to new hardware as I write this. Fundip and all its sites should be back up soon!

We apologize for the inconvenience this causes, please contact support if you continue to have issues after this post has been resolved.

Happy DreamHost “The Lick’em Stick was always the best part” Team!


2:54 am

MySQL server maintenance April 17th@2am Pacific – Done

Posted (April 15th, 2009 at 2:54 am PST) by Ken

The following  MySQL servers will be down for maintenance at 1am April 16th Pacific:

flipper, fin, nobnin, riffraff, and morris

The servers should only be down for about 10 minutes up to a maximum of 30 minutes.

Status will be updated when we start and when the services are back up.

UPDATE: Due to another issue, the mysql maintenance will be deferred until further notice. Currently, the Mysql servers that this affects are still up and fully operational.

UPDATE: Maintenance has been rescheduled to April 17th at 2am Pacific.

UPDATE: Starting maintenance. Should be finished by 2:30am Pacific. mysql machines will be down until then.

UPDATE: Maintenance is done. All mysql services have been restored on these machines.


5:00 pm

Elusive issues on Trafficante

Posted (April 14th, 2009 at 5:00 pm PST) by JeffH

As anyone on this particular server has noticed, Trafficante hasn’t been performing well at all in the last couple days with high load, fairly significant downtime, and poor performance when it’s up.  Our admins are currently focusing a good deal of effort on isolating and resolving the problem, but the issue has been difficult to pin down so far.   We can’t do a hardware replacement, as the issue could very well still be one of software, and the problem may very well follow over to the new hardware.

We are continuing to troubleshoot this, and will post updates as they become available.  As always, feel free to vent here, but if you have question you’d like answered, file a support ticket!   We appreciate what patience everyone can muster while we sort this one out!

Update!

We are currently moving half of the users from the machine so as to permanently lighten the load  and restore stability.  There were initially some issues with the moving script which have been sorted, and the moves are progressing…there is a lot of data, but the situation should improve steadily up until the moves are complete.


3:15 pm

Network Maintenance Tonight

Posted (April 13th, 2009 at 3:15 pm PST) by jeremy

We were just alerted by Comcast that they will be taking down one of our circuits between 2am and 3am on 04/14 (tomorrow as of the time of this writing) for emergency maintenance. We expect zero impact on the network since we were running without them until last week and have plenty of extra capacity. However, we’re making a maintenance window lasting until 7am as a precaution.


8:11 am

fairfax hanging

Posted (April 12th, 2009 at 8:11 am PST) by Kyle

fairfax is having trouble being power cycled remotely, with remote hands an hour away from the data center we have dispatched our own personnel to address the issue. Expected downtime is 30-40 minutes.

Update: the machine has been manually rebooted and is now backup up and running!


6:14 pm

Panel Down

Posted (April 10th, 2009 at 6:14 pm PST) by Ralph

Our central database server is having problems so we have to shut down our panel as we address that issue. Sorry about the problems with causes you with making changes to your account. This does not affect service uptime, it is simply a web panel issue.

Update: Panel is back up but we will be watching our databases closely to make sure no further problems occur.


7:45 pm

Nitti Fried – RESOLVED

Posted (April 8th, 2009 at 7:45 pm PST) by Patrick

The shared hosting server, nitti, blew it’s raid card, rendering the server unusable. Data is being restored from backups now to new hardware and all sites will be working again shortly.

Please check this space for updates.

– UPDATE Wed Apr 8 23:01:11 PDT 2009 –

Unfortunately, the “new server” was misconfigured and had a malfunctioning network interface. This slowed down the process considerably. Please accept my apologies. A second (fully working) replacement server is being configured now. Once again, all sites will be working again shortly (promise).

– UPDATE Thu Apr 9 01:28 PDT 2009 –

The new server is now up!  Some configurations are still running at this time, but everything should be back up and running now.  If you continue to experience any downtime with your services, please contact Support and they’ll be happy to check it out for you.

– UPDATE Thu Apr 9 03:23 PDT 2009 –

We are aware of the “503” errors that are coming up on some sites and are running updates to the server to resolve them.  Feel free to contact Support about this, as that will help to identify any remaining issues.  But, these errors should be cleared up shortly!

– UPDATE Thu Apr 9 05:00 PDT 2009 –

The “503” errors should all be resolved now!  Please contact Support if you still see any issues!


1:54 pm

(Success!) Network Maintence this Friday April 10th

Posted (April 8th, 2009 at 1:54 pm PST) by DreamHost

This Friday, April 10th at 11:45 pm Pacific time we will be doing some network maintenance. This work is expected to be transparent to end users. It should last around 2 hours while we shuffle our internet links around on to a new router to add more horsepower. We have more than enough bandwidth to cover any single link being down while this is happening, so no one should notice any problems.

If anything goes horribly wrong, all services will be affected except this blog! It will be updated with information if anything happens.

Edit: 1:45 AM Saturday April 11th Success! We migrated over 6 links to a new router and all is working hunky dory near as we can tell! Please contact support if you are having any problems.


8:21 am

snowbird snowed in

Posted (April 8th, 2009 at 8:21 am PST) by Kyle

Snowbirds network card went on the fritz and was moved to new hardware, all services have been restored. If you are still having issues please contact support.


1:16 pm

Panel Downtime

Posted (April 7th, 2009 at 1:16 pm PST) by David

It looks like we may be having problems with our central databases again. It’s effecting our control panel, WebMail and DreamHost Apps. We apologize for any inconvenience this may be causing. The admin team is looking into this now to try to get it resolved as quickly as possible.

Please stay tuned for updates.

Update: 16:55 April 7th: First off we would like to apologize for the recent database errors we have been experiencing that cause your panel access to go down. We are getting all our heads together to try to resolve these central database issues once and for all. We have stabilized the central database machines now and we are working on finding the source of these issues. All services are up and running currently but we are watching the database closely. Once again we are really sorry about the problems this has caused you and we aim to have this repeat issue completely resolved as soon as possible.


 
© 1996-2009, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
10:18 am

Missy MySQL Server Downtime

Posted (April 6th, 2009 at 10:18 am PST) by Justin@DH

The missy MySQL server is having a hardware issue, one of our admins is on his way down to the data center now to diagnose the problem. We are working to get this resolved as soon as possible. When we have further information to share we will update this post. The missy mysql server is used by “spacey” cluster customers only, no other clusters are currently effected.

— UPDATE —

The admin has diagnosed and replaced the hardcare causing the problems, and the missy MySQL server is working normally again. If you do continue to have any issues related specifically to this server please contact support and provide details about the problem. Again our apologies for the downtime.


6:17 pm

Crenshaw downtime

Posted (April 5th, 2009 at 6:17 pm PST) by andrea

The web server Crenshaw is being moved to new hardware due to its bad NIC (network interface card). The admins estimate the process to take about 30 mins. We’ll update you here as soon as things are looking batter for that machine. We apologize for all inconvenience this is causing.

If you’re not sure this is what’s affecting you, please check for your account information to find out the web server you’re on. You can find that under Account Status in the upper right corner of your web panel.

Update 8:45pm PST: The server was back up and running within the expected time frame, it just took us to get everything running again a bit longer. Sorry about the delay. All services are now operational. If you’re having any more problems with your domains on Crenshaw, please contact support for more assistance and include as many details as possible.


7:28 pm

Panel Downtime

Posted (April 4th, 2009 at 7:28 pm PST) by micah

Our central database went down about half an hour ago causing the panel to be inaccessible for a short while.

The database is back up now and I’ve restarted the panel webservers. Everything should be back to normal but if you’re still experiencing issues pleast contact technical support.


5:49 pm

Network issues

Posted (April 3rd, 2009 at 5:49 pm PST) by Terri

We are currently experiencing networking issues. This is affecting some machines, as well as mysql services and oneclick installs. We are troubleshooting the issue and will fix it as soon as possible. We apologise for the inconvenience.

Update: Thankfully the network issue was resolved in a short period of time. There was a misconfiguration in our core router, and our network administrators patched it up.


3:02 pm

Backup Redux!

Posted (April 1st, 2009 at 3:02 pm PST) by micah

While we’re still in the process of restoring everything and getting the data moved off of the suspect RAID arrays, we think we might have recovered almost all of the backup data we previously thought that we would not be able to recover as seen here:

http://www.dreamhoststatus.com/2009/03/07/backup-server-problems/

We are in the process of copying the data off of the degraded array so we may be unable to recover some files, but getting the array back online gives us the hope to recover a large amount of the data. We will not be copying the data over any backups you have restored. We will be providing instructions on how to get your previous data via email. Thanks go to our hard working NOC employee Sandon for sticking with this issue and getting us to a point where we can recover this data.

We sincerely apologize for the problems this backup service crash caused you. Our current ETA for the restore is still going to take about a week or two since this is several TB of data.

Update April 7 21:00: The data recovery is still coming along. We have received complaints about the stability of the backup service unfortunately. All of our backup admins are currently working on the data move and unfortunately we will not be able to fully address the stability issue on the new service. We are also dealing with a problem of disk space availability. The failed raid left us short handed as far as space for data is concerned. We are expecting to have more space by the 17th. Sorry about this backup service not being the service you expect just yet.

We have moved backup information to the following post: http://www.dreamhoststatus.com/2009/04/16/backup-service-update/


5:53 pm

DreamHost Database Having Issues

Posted (March 30th, 2009 at 5:53 pm PST) by Terri

One of our central databases is having issues right now. DreamHost administrators are troubleshooting the issue right now. We will update this space with more information as it becomes available. This affects:

Panel
Webmail (Roundcube and Squirrelmail)
DHapps

Update: And we’re back up and running. Just a small hiccup in our network took things down. We’re sorry for the inconvenience.

Update: Looks like we spoke too soon. The database is overloaded at the moment. We’re attempting to ease the load on the database while we look into the issue.


2:22 pm

Homie mail problems

Posted (March 29th, 2009 at 2:22 pm PST) by danwy

We are currently experiencing some issues with the service that updates mail accounts on our homie server cluster. This will affect recently moved accounts and any new email addresses that are created via the web panel. We’re working on resolving this issue now. We don’t have an ETA on when this will be fixed yet, as there are a lot of records to update. We will post more info here as soon as we have it though. We’re sorry for any inconvenience this causes you.

Update: 6:48pm PST The issue is actually with the saved paths for recently moved and new mailboxes. Some of the server records are pointing to an incorrect server and we’re in the process of running an update (which has been going since the initial post was made) to correct all of these. Mail coming in to these addresses with the broken paths will not be lost, they are sitting on our servers waiting for the paths to be updated. At which time, mail should be delivered. We’ll resolve this post once the update has completed and we’re sure that things are running smoothly again.

Update: Mar 30 2:30pm PST We’re sorry for the lack of updates, but there is no new information regarding this issue. Our updater is still running 24 hours later and we’re kind of at a stand still until that completes. According to how much has been completed up to this point, we’re guessing that it is going to finish sometime late tonight or early tomorrow morning. As soon as we have any more details regarding this issue we’ll be sure to post them here.

Update: Mar 31 10:30am PST The updates finished up a little while ago and for the most part everything should be back to normal. If you continue to have issues with accessing your mail after this point, please contact support and we can investigate the issues on a case by case basis now. Again, we apologize for any inconvenience this caused you.

Update: Apr 2 11:35am PST Unfortunately, the problem has resurfaced last night. We resorted to using the last good status for the email addresses on this server cluster while looking into a permanent solution. This means that any newly created addresses (including those on the accounts that were just moved to the new homie cluster) or changes to addresses made as of yesterday morning are pending. Sorry about the continued inconvenience. We’ll post an update as soon as we can.

Update: Apr 2 5:55pm PST Progress has been made and we believe that all email on the homie cluster is now current. We’re leaving the case open however, to make sure that the same issue won’t poke its head up again soon after the updates have been made. We appreciate your patience, and we’ll update this post as soon as there is a solid and final resolution.

Update: Apr 5 10:35am PST We’ve been monitoring this situation closely for the last couple of days and we feel confident that the underlying issue causing these problems has been resolved. There may still be some customer mailboxes that need to be updated still. If you have already contacted support in regards to this and have not received a reply yet, we are working as quickly as we can to respond to you. If you are experiencing these issues with your maibox on this cluster and haven’t contacted us yet, please do so.


6:24 pm

Robin laid an egg

Posted (March 23rd, 2009 at 6:24 pm PST) by Jason C

Users on robin will be seeing issues with their sites as the server is currently running into kernel errors. Our admin has dashed over to the datacenter to hopefully remedy this issue quickly, but we’ll do our best to keep you up to date here.

Update Mar 24, 2:05PST: Robin is available again. Sites hosted from it should be working now.


1:03 pm

dumdum getting a new stick

Posted (March 23rd, 2009 at 1:03 pm PST) by kitchen

The machine ‘dumdum’ in the spunky cluster is having some software difficulties, so we’re going to move it to new hardware with a fresh install of the OS. This process should take about an hour and we will post an update once everything’s all set.

Thank you for your patience!

Update: Sorry for the long delay, but dumdum is back up. Took quite a bit longer than I thought it would! Please let your friendly DreamHost technical support technician handle all of your remaining technical queries.


12:11 pm

Homie mail problems – disappearing mail/inability to log in

Posted (March 17th, 2009 at 12:11 pm PST) by Ralph

We have discovered a problem with some mailboxes in the homie cluster first becoming inaccessible, then losing data once the mailbox is restored. We do have backups of your mailbox but most recently received messages may be lost. We are tracking down the cause of this issue then we will restore what backups we have. Sorry about the problems this has caused you. So far it seems that mail users on Bruno are most affected. If you are not on Bruno please contact support for further assistance with this issue. If you can use that status post title as your subject for support that would be great.

UPDATE (6:25 pm Pacific): We have created a tool to fix all accounts affected by this issue. We believe we have found the source of the problem. We will continue to work on the source of the issue but everything customer facing should be resolved now. In some cases you may have updated your user to try to get it to work, those accounts may have been missed. So if you are still experiencing problems please contact support so that we can get your mail restored.


 
© 1996-2009, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
6:04 am

Homie and Janky downtime.

Posted (March 17th, 2009 at 6:04 am PST) by mir

We apologize for the inconvenience this morning – a number of sites have gone down because their associated apache server died. Fortunately the fix is easy (we are restarting all of the apaches now) but we still do not have the cause determined (we believe it is related to changes in log rotation). We’ll update this post once we have more details.

Update 9:57PST: Apache was upgraded across janky and homie last night and configuration updates were scheduled for 5:45 this morning.  The updates reloaded apache but in some cases it didn’t bode well.  The issue was quickly corrected ~6:30ish so if you still saw problems after that then they were likely unrelated or a straggler that still needed some TLC.  At this point if you are still seeing issues please contact support.


7:06 pm

Changes to global /etc/wgetrc

Posted (March 16th, 2009 at 7:06 pm PST) by kitchen

Short story: we’re adding continue = on to /etc/wgetrc on all of our machines.

Long story:
Folks like to use wget in their cronjobs. Understandable, as wget is a really great way to grab a file via http via the command line. Unfortunately, wget has some fairly annoying default behavior.

Say, for example, you have a cronjob set up to run hourly which calls wget to grab cron.php from your domain example.com:
@hourly wget http://example.com/cron.php
Simple, right? Well, the problem with this is, the first time this runs, it will create a cron.php in your home directory. The second time it runs, it will create cron.php.1 in your home directory. Third time? cron.php.2 … and so on. Before it can create cron.php.123543132 (if that’s how many times your cronjob has run) it has to try to create all of the ones before it, which can cause all sorts of havok on our servers. Not to mention the stale files full of garbage just laying around.

Since wget appears to not have any explicit method for disabling this behavior, we’ve been experimenting with some of the various settings in /etc/wgetrc to make sure that we are able to stop this behavior with little impact on customers. Our experimentation has concluded that the safest option to use for this purpose is the ‘continue’ option, and because of that we’ll be setting this to ‘on’ in /etc/wgetrc on all of our machines over the next week or so.

If this change causes you any problems, please let us know right away and we’ll take a look at it. If you think you know better than we do (yea, right, right? ;)) and have a better option for us to use, let us know, and we’ll look into it!

The Happy DreamHost Beat Wget Into Submission Team


7:54 pm

Issues with our Homie cluster

Posted (March 14th, 2009 at 7:54 pm PST) by danwy

We are currently experiencing issues with one of our server clusters, homie. If your site is on this cluster and it is currently down, this is the reason why. Our admin team is aware of this and is on the phone with our datacenter right now trying to find the cause. We don’t have any additional information at this time, but we will update this post as soon as we have more specific details. We apologize for any inconvenience this causes you and we hope to have everything back up shortly.

Update: Sorry we didn’t include this earlier. Here’s a list of servers that may be experiencing issues:

barker bonanno castellano castiglia genovese mangano mcgurn rothstein scalise schultz spillane spilotro

Update: This issue was caused by a power strip failure in one of our server racks. We have the datacenter engineers on their way to the datacenter to resolve the issue. We don’t have an ETA on when things will be fixed, but we are doing all that we can to remedy this issue as soon as we possibly can.

Update: For the most part, this issue has now been resolved. The datacenter engineers were quick to respond and resolve the problem with the circuit breaker. There was a small handful of servers that weren’t booting up correctly, so we had to go through and manually reboot them. We have most of the services back up and running, but there are still a few PS servers that we’re still working on. We should have those all back up and running shortly. We’ll post one final update once we’re confident everything is resolved.

Update (1:39am): Alright, whew. Everything should be fine now. What actually happened was a circuit in the datacenter blew and it just so happened that the switches that your machines are connected to were out because of that. It took some time to get a tech from the datacenter in there to reset the breaker and then after that I had to go and manually check each machine. The DreamhostPS and PS Mysql customers had some networking issues that also slowed things down. I apologize for the downtime! If you’re still having trouble, please send a message to our support staff.


11:07 am

Quick network cutover

Posted (March 12th, 2009 at 11:07 am PST) by DreamHost

Tomorrow night at 8:00PM Pacific March 13, 2009 we will be doing a quick network cutover. We are moving our of some older racks in to newer purpose built networking space. The cutover should be nearly transparent, with at most a minute or two of downtime.

This affects all services for people in randy, frisky, swarthy, looney, blingy, and spacey clusters.

— Update —

The cutover went extremely well. All services are up and responding. If you are having any problems whatsoever, please contact support.


11:00 am

Email login changes

Posted (March 9th, 2009 at 11:00 am PST) by jordan

** Originally posted on December 2nd 2008 **

Starting January 7th, 2009 we will stop allowing customers to login via mailbox names (eg “m1234567″) . Instead you will need to change that login to the full email address assigned to that mailbox. This will affect all customers including dedicated and Dreamhost PS. For information on configuring your email client this way please take some time to look at the Wiki article linked to at the bottom of this post. If you are a new Dreamhost customer, chances are you are already using your full email address to login so you will not have to make any changes. Next week we will start e-mailing customers that we see logging into mail servers with their m1234567 logins. It will include the address that you should be logging in as, as well as the mailbox that you are currently using to login. If you have any questions regarding the email address you should be using to login please contact technical support and they will be happy to help you.

Also, if you have any scripts/php/cgi that use SMTP authentication to send e-mails make sure that they are updated as well to use the full email address for sending their messages.

http://wiki.dreamhost.com/E-Mail_Client_Configuration


9:21 am

Backup server problems

Posted (March 7th, 2009 at 9:21 am PST) by danwy

Some of you with backup only users may have noticed some issues with being able to connect or see your files. It seems that the backup server is having some disk issues. Our admin team is currently looking into the issue and hopes to be able to resolve the issue without any data loss. Please keep an eye on this post, as we’ll make updates on the situation as soon as we have them. We’re sorry for any inconvenience this causes you.

UPDATE (March 10) – We are currently in the process of contacting support for the hardware vendor. This is unfortunately a slow process. At this time we are not optimistic about 100% data recovery, we are expecting to recover possibly 80%, but we are still doing everything we can to try to recover all the data. Unfortunately information has been slow to come by, but we’ll do our best to update this post with any new information we have.

UPDATE (March 16) – Sincere apologies for the length of time that has been allowed to slip by without an update. Our goal is to have the backup service in a working state tomorrow, however we still do not have our hands on the data that has been lost due to the previous outage. At this point our goal is to restore the service, if not the data. We will continue to work with our vendor on the failed volume issue and attempt to get the data off.

UPDATE (March 17) – Unfortunately we have still gotten no where in our attempts to recover the data. At this point we are calling the backup data a total loss. Sorry for the problems this has caused everyone using the service. We will be cleaning this up and getting in touch with you if you sent in a support request. Unfortunately we were unable to get the service restored today as planned so the goal has been pushed to tomorrow.

We have moved backup information to the following post: http://www.dreamhoststatus.com/2009/04/16/backup-service-update/


6:52 am

Cloak server hardware issues

Posted (March 6th, 2009 at 6:52 am PST) by patrickm

Unfortunately, the Cloak server lost a hard drive this morning!  We sincerely apologize for the downtime this is causing.  The RAID controller on the server also started reporting errors related to the failed disk pair.  So, rather than risking the loss of data, we have replaced the RAID controller and are moving data as quickly as we can.  We have also placed this server on to a gigabit switch to speed up that process.

Unfortunately, we are yet unable to provide an estimate for when this will be completed, but our engineers are working diligently to resolve this issue and bring the server back online.  Please check back here for updates, as they become available!

Update 1:10pm pacific: The new machine is up and running from backups, however we’re going to take it down for a few minutes to replace the raid controller in it. When it comes back up we’ll begin the process of restoring all of the data from the backups to the local machine. Your data will have rolled back 24-48 hours.


12:43 pm

Terranova downtime : Resolved

Posted (March 5th, 2009 at 12:43 pm PST) by mir

What was supposed to be an extremely brief reboot of the server for maintenance has resulted in more severe downtime as the server is not coming back up like it should be. The admin team is investigating and should have it back up shortly. This post will be updated as soon as we have an ETA or the server is back up (whichever is quicker).

Update: It’s back up, we just had to get someone to physically restart the box and it came right up but we do apologize for the inconvenience.


9:30 am

Randy cluster power failure

Posted (March 3rd, 2009 at 9:30 am PST) by Ralph

The randy cluster, also home to the previous spoiler issue, is unfortunately having problems due to a failed power strip. Our administrators are actively working on this issue. More information should be posted soon. You can tell if you are in the randy cluster by looking at the ‘Email Server’ value in the ‘Account Status’ from the DreamHost web panel.

Update 9:35PST: Two circuits failed, the strips were not the problem, we are working with the datacenter staff to get them back online in an expedicious manner.

Update 10:05PST: Power has been restored to the failed circuits, service has been restored to the affected servers.  If you are having trouble with your site still it is unrelated to the power outage, please contact support via the panel.

Update 11:00PST: We have evaluated this downtime and unfortunately this issue was not in our control. The problem was with three circuits that are managed by the data center.


8:49 am

Spoiler lives up to its name

Posted (March 1st, 2009 at 8:49 am PST) by DreamHost

Spoiler, one of our file servers, stopped responding and isn’t serving up files anymore.  This will cause web service outages on the following servers:

arkanoid bob boppity caesar deblume defender dinar dinero ellipsis euro fidget forint franc gage galaga gap go grizli guilder happy ikari ike intheroo ivan joust katamari klax korben kroner limbo-randy mark marvin nimitz obiekty24 orbital paperboy patton peso pound ps5500 ps5643 qbert quid replicated researchdirector rygar scagnetti schilling scipio shekel slaughter tetris ultima wasabi yen yoda zorg

Someone’s already been dispatched to the data center and we’ll get this back up and running as soon as we can.  The expected downtime is 30-45 minutes.  Sorry everybody!

UPDATE (10:49 am Pacific): Looks like some of the networking hardware has failed and needs to be replaced, so this is taking longer than expected.  Folks are working on this as quickly as they can and I’ll update this post as soon as there are any new developments.

UPDATE (1:42 pm Pacific): The hardware has been fixed. Right now, the drives are being checked for integrity before the machine can be booted back up. Unfortunately, this will take another few hours. Very sorry about the continued inconvenience. We’ll update this post as soon as the drive check is finished.

UPDATE (5:51 pm Pacific): Unfortunately, bad news: the file server came back up, but then it froze again. Our admins continue working on a resolution. We appreciate your patience and understanding.

UPDATE (10:38 pm Pacific): Things are looking up, we’ve got the file server back online and are fixing the mounts to the webservers now. There will be another update here when all is well.

UPDATE (1:40 am Pacific): Got the final word from the Admin team that this issue is now resolved!  If you are seeing any errors (such as “bad_httpd_conf”) on your websites, or any other issues, please contact Support and we’ll get those fixed up for you!

UPDATE (March 5th): Sorry about the extended problems some of you have been dealing with. If you are still noticing problems we will need to hear from you in order to get your site working for you. During the file server outage we had to lock our configuration files and exclude users with their data on the file server spoiler. Once spoiler was back up we removed these blocks and reconfigured our web servers. This fixed most of the problems but there were still others that had to be addressed directly. So if you still have problems with bad httpd conf or other issues please contact support for further assistance.


 
© 1996-2009, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
12:45 pm

DNS slowness/lookup failures

Posted (February 26th, 2009 at 12:45 pm PST) by Sandon

Our DNS servers are currently running slowly/failing to do lookups. This is causing some of our sites to resolve slowly or not at all. Our admin team is working on the issue right now. More updates to come.

Update: We are experiencing a DNS flood to our primary DNS servers which has stopped for now but the admin team is continuing to investigate so that we can attempt to avoid and further issues related to this incident. We apologize for the inconvenience!

Update: We have found the source of the problem. It looks like there is a windows trojan going around that attempted to make contact with a domain that is pointing to our nameservers, but isn’t actually being hosted by us (was likely an old customer that removed it from hosting, but didn’t change the nameservers to point away from us). This caused the flood of traffic to come in and overwhelm our DNS servers. You can find more information about this trojan here:

http://www.sophos.com/security/analyses/viruses-and-spyware/trojsecondtaa.html

Unfortunately, DNS attacks are nearly impossible to stop if they’re well crafted. In this case, all of the traffic is perfectly legitimate and there’s nothing special about the packets that any protection system could really block. We are in contact with verisign (the registrar of that domain) now to try and stop this domain from pointing to our nameservers and are doing everything we can on our end to resolve this issue.

Update: This issue should now be resolved, as we’ve been able to filter out traffic for the offending domain, and send it to the bit bucket! We haven’t seen a surge in DNS traffic since 02:20pm PST, so things are in the clear!

Update: Unfortunately, we are still being attacked sporadically even after blocking this virus and are still investigating the issue. We are not the only ones being affected by this. Time Warner cable is also suffering from this as well:

http://arstechnica.com/security/news/2009/02/time-warner-cable-blames-ddos-attack-for-spotty-service.ars

We’re doing all that we can to resolve this issue, but since it’s out of our control there’s only so much we can do.


9:33 am

Joker stabillity issues being worked on.

Posted (February 26th, 2009 at 9:33 am PST) by mir

I am terribly sorry for the issues customers have been having with this server, particularly when most of the users on it have been moved there in hopes of achieving improved performance. We’ve been investigating it but we haven’t properly posted our findings so that is the purpose of this message and we’ll update you here until we have things resolved.

So far we’ve looked for bad users (and have in fact removed a few which helped). Unfortunately that was not sufficient as there was more going on. As it turns out the machine was running out of file handles which was then causing everything to back up – one of our admins corrected this issue yesterday and everything was looking vastly improved but it appears that problems are back today. We’re waiting to hear whether this means there is another issue or if it’s just that the previous fix was not sufficient and needs extra attention. Once we have more details we will update this post.

Update: it did turn out to be some bad scripts under a pair of users that were causing the more recent load issues – we’re still keeping a close eye on it but so far the load is looking significantly better and the machine is more stable.

Update: while it’s remaining stable for longer periods of time we’re still seeing load spikes and continue to investigate.

Update: after stopping a few DoS attacks yesterday the server appears to be quite a bit more stable – the load is generally under 10 or topping out in the mid to high 20s (which on this machine is actually around 4-15 load as it has double the hardware of our older servers). That said it did spin out of control once today so we’re going to continue to watch it and clear out any lingering problems (things are vastly improved already though).


2:39 am

Some servers showing errors.

Posted (February 26th, 2009 at 2:39 am PST) by DreamHost

The following servers are having some issues that are currently being investigated:
sabertooth wonderwoman hawkman bishop mystique hobgoblin

Update 05:53 AM PST: Hobgoblin is back up and serving content. The other 5 servers should be back up within 10-15 minutes. More updates to come.

Update 06:22 AM PST: All services on all the above servers should be restored. If you are still having issues accessing your site please contact support.


9:48 pm

File Server Trouble

Posted (February 24th, 2009 at 9:48 pm PST) by Terri

One of our file servers has crashed. We are looking into the issue and we will hopefully have it back up and running within an hour. We will keep this space updated as we progress.

Here are the machines that could potentially be affected:

7up amp arizona bambi barqs boba cactuscooler calpico chai clamato codered crush crystallight dads dasani dietrite eggnog espresso fiji frappe fresca fuze gatorade geyser gingerale grapico hi-c hiball horchata inko jolt koolaid lipton martinellis milk mrpibb mug nehi nesbitt nestea niagara oj orangebang pellegrino pepsi perrier powerade propel punch refresco riptide rockstar rootbeer schweppes seagull seltzer silk simpledollar snapple sprite squirt sunkist sweetums tea tizer tropicana v8 worldtown yoohoo

This only affects web services for these machines.

Update 11:27pm: Unfortunately this is taking longer than anticipated. It looks like this may take another hour or two.

Update: 12:49 AM PST: The file-server is back up and serving data. Please contact support if you are still having issues accessing your site and you are on any of the above servers.


12:25 pm

Panel Temporarily Unavailable (round 3)

Posted (February 24th, 2009 at 12:25 pm PST) by David

Looks like the central database servers are at it again. The control panel (https://panel.dreamhost.com/) as well as the WebMail interfaces (SquirrelMail and RoundCube) are currently down (temporarily). Our database guru is investigating the situation now and hopes to get it back up again as quickly as possible. We’ll post updates as soon as more information is available. We apologize for any inconvenience this may be causing.

NOTE: This will not effect your web servers, private servers or non-webmail e-mail accounts.

ADDENDUM: one side effect of the central database issues we have been having was the temporary loss of a large portion of our cloaking data table. Our admin team is in the process of restoring that information right now and expects to be done in the next couple of hours (2 PM PST was the original target but we just had another outage so we’re not 100% sure on when this will be completed). The good news is that you are not required to wait – if you prefer you may add the service back yourself via the panel here:

https://panel.dreamhost.com/index.cgi?tree=domain.manage&

and it should come back up once the DNS updates.

The DreamHost “we hate databases as much as you do” team!

UPDATE 1:10pm PST: The central database is back up again (as well as the control panel and WebMail interfaces. The admins are reviewing what they’ve found but don’t have a detailed update at this time.


2:09 pm

Panel having issues

Posted (February 20th, 2009 at 2:09 pm PST) by Terri

The WebPanel just began having issues. We are on top of it and diagnosing the issue. This should be temporary but we will keep this space updated. This outage will also affect webmail (Squirrelmail and Roundcube) as well as DHapps.

Update: We’re back up, and we’ll have more troubleshooting info very soon!


5:56 pm

Panel offline tonight, 10pm PST

Posted (February 18th, 2009 at 5:56 pm PST) by DreamHost

We will be taking the panel offline (intentionally!) tonight, Wednesday February 18, starting at 10pm PST to attempt to address the database issues of the past two days. We expect to have it back online by 11:59pm PST.

Update: 11:05pm PST, all back online.

The modifications and upgrades went well. The debug level was also changed to provide more information in case of another crash. Hopefully there won’t be any further crashes, but if there are this should help us troubleshoot and resolve the problem more quickly.


4:46 pm

Backup Machine Being Worked On

Posted (February 18th, 2009 at 4:46 pm PST) by Leon

Sorry for the short notice everyone but we’re bringing down one of the backup machines that your backup users are housed on for backup.dreamhost.com. The estimated downtime for it about 30 minutes but we’ll hopefully have it up sooner than that!

Check back for updates!

Update 4:55 PM: Wasn’t as long as we expected and everything is back up including all of your backups! If you notice any issues, please contact support.

– The Happy DreamHost Mid Day Maintenance Team!


3:58 pm

Craigslist is blocking messages sent from Webmail

Posted (February 18th, 2009 at 3:58 pm PST) by andrea

Craigslist has been blocking messages sent from DreamHost webmail for the past week or so. We have contacted them a couple of times since, asking them to whitelist us, or provide us spam evidence so that we can take action. We haven’t received an answer yet. They have been much more responsive in the past, so we’re hoping to get this resolved asap as well.

We apologize for the inconvenience this is causing. It’s only affecting webmail sending, all other ones are fine. Please check back here for updates.


3:39 pm

Digdug done died

Posted (February 18th, 2009 at 3:39 pm PST) by Leon

Unfortunately, digdug is not coming back from several reboot attempts so we’re moving it to new hardware. Should be back in about an hour!

Update 6pm: Ugh, there’s been many issues with hardware that we’ve tried to move digdug over to. We’re still working on it and hoping to get it back up within another hour.

Update: 6:42pm: Okay, digdug is back up. Terribly sorry for that long downtime! If you’re still having issues, please contact your friendly DH support robot.


 
© 1996-2009, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
10:26 am

Panel Temporarily Unavailable (round 2)

Posted (February 18th, 2009 at 10:26 am PST) by David

Our central database that controls panel has crashed again and is recovering right now. This is a very large database so it can take some time, but we are estimating that it should complete in an hour or less. We will update this post as things progress. Hopefully our experience from yesterdays outage will help us get this back up more quickly this time. We apologize for the inconvenience!

NOTE: This also effects access to our WebMail interfaces (SquirrelMail and RoundCube) and DreamHost Apps sites.

UPDATE: 11:28am PST It’s back up again! We’re keeping a close eye on the database server for a while though, just in case.

UPDATE: 12:16pm PST Looks like it’s down again. The admins are investigating. Stay tuned for more updates.

UPDATE: 12:40pm PST It’s UP again.

UPDATE: 12:57pm PST It’s down again. Admins are still investigating the cause of these outages.

UPDATE: 1:15pm PST It’s UP again.

UPDATE: 1:30pm PST Basically what’s happening is we’re 99% sure the memory (RAM) issue has been resolved. However the database process keeps dying without any useful information as to why. Fortunately however it seems to recover automatically in about 10 minutes or so without any data corruption (which is awesome). Our top database guru is on the case monitoring it very closely to try to figure it out and get it stable again. Again, we sincerely apologize for any inconvenience this is causing. We’re working as quickly as possible to get this problem resolved once and for all.


12:05 pm

Panel Temporarily Unavailable

Posted (February 17th, 2009 at 12:05 pm PST) by Terri

Our central database that controls panel has crashed and is recovering right now. This is a very large database so it can take some time, but we are estimating that it should complete in an hour or less. We will update this post as things progress. We apologize for the inconvenience!

NOTE: This also effects access to our WebMail interfaces (SquirrelMail and RoundCube) and DreamHost Apps sites.

UPDATE! 3:28pm (PST): Unfortunately, this is still an issue. After the first reboot, it started working fine, but then promptly died again. The errors we were receiving indicated a memory problem, so we swapped out the RAM and rebooted (and needed to rebuild again). This didn’t fix the issue though, at it crashed again about 15 minutes after coming back up. Unfortunately, we’re in the process of re-diagnosing the problem and don’t have an ETA on when it will be back up. This does have our admin teams full attention though and they are doing everything they can to get this back up and running as soon as they possibly can. We’re sorry for the inconvenience this causes you.

UPDATE! 5:29pm (PST): For those of you who can’t get in using webmail, please note that IMAP and SMTP access should still work. This means that setting up a desktop email client (such as Postbox, Thunderbird, Mac Mail or Outlook) is a perfectly acceptable workaround.

UPDATE! 8:17pm (PST): Okay, the database has been recovered and is plugging along. Our admin team is staggering the services that they’re bringing back up as to not overload things. First up is the web panel and signups. After about 15 to 20 minutes, we’ll be bringing up webmail. Another 15 to 20 after that, easy one-clicks and DreamHost Apps should come back up.  Once it’s all resolved, we’ll make sure to post a final update here.

UPDATE! 8:44pm (PST): All core services should be back up and running now. If you sent in a message while things were being the opposite of awesome, expect a response shortly. Should any problems with our core services manage to rear their ugly heads — and after today, we really hope that they don’t — contact support and give us the full output of any panel, webmail or one-click errors you might see.


6:16 am

Lira died

Posted (February 14th, 2009 at 6:16 am PST) by Sandon

It looks like the web-server ‘lira’ has died and is in some sort of reboot loop. It power’s for a second and then turns back off, then powers up again in a loop. I am moving this machine to new hardware now and it should be back up within 1-2 hours. I will update this post when it is back up and happy.

Update 08:05 AM PST: Ok, Lira is up and happy again on new hardware! The good news is that the old machine was a dated pentium 4 with 3 GB of ram and the new machine is a brand new quad core xeon with 4 GB of ram so it should be a significant upgrade for all the customers on lira. Please contact support if you are still having any issues accessing your site and you are on ‘lira’


1:20 am

Postal file server issues

Posted (February 13th, 2009 at 1:20 am PST) by kitchen

We’re currently experiencing some trouble with one of the file servers serving the postal cluster. The file server blew a drive and is currently rebuilding its array. While the system is still serving files properly, it’s doing it a lot more slowly than it normally should. Unfortunately, due to the nature of this process, we cannot stop it without risking data integrity if another drive should die while this one is rebuilding, and the file server is already configured to rebuild at the lowest possible priority. The process is 38% completed and we expect that it will finish in a few hours.

The list of machines affected by this are as follows:

abe amw annie apu arnie audrey barriodelcarmen bart beroe bitters blatant bloodcopy bomberstudios brandine brockman browser burns carl cdeef chalmers cheeky clancy cletus concrete daigiavip dragonball drederick duffman evangel ewaste1 flanders ga gaz gezenbilir ghost gimli gir glass gloop gnif gotcha harpo hibbert hmtotc homer hyperblocks jasper jebediah jimbo johnmccain jubjub kearny krabappel krusty lablogs larry lisa lovejoy manjula mbr mccallister memmo mendelsohn millhouse mita mona monkey mtrends mylifenavigator mylove8x ned nelson noticiaria osteele otto pappgabor patty patty1 ponderous porker ps3845 ps3927 ps4112 ps4208 ps4326 ps4419 ps4473 ps5248 ps5389 ps5546 ps5624 ps5658 ps6194 ps6300 ps6428 ps6465 ps6803 quimby raiden ralphie rhythm salt sanjay scooter seamus shemp skinner softwa syscount tak tastybooze tdz touchofdeathforums tracer vajjsistas vivartes vpg webprojnet websaves wetdevil willie yerba yggsdrasil zim

Mail users on the postal cluster will also likely experience slowness related to this file server.

We will update this post once things are back to normal. Thank you for your patience!

– The Happy DreamHost Late Night Storage Team

UPDATE: Everything has been resolved. The drive rebuild has completed. We will continue to monitor the stability of the fileserver. Again we sincerely apologize for the inconvenience this has caused and appreciate your patience.


4:38 pm

Cornelius Moving to New Hardware

Posted (February 12th, 2009 at 4:38 pm PST) by Leon

Cornelius is currently experincing issues coming up from attempts at a reboot, so we’re going to move it to new hardware. It should be back up within a few minutes.

More updates to come!

Update 5:30PM: Cornelius is now back up on its new hardware and seems to be chugging along nicely. If you’re still experiencing any issues, please contact our support team!


10:21 am

Mail delays on swarthy cluster

Posted (February 12th, 2009 at 10:21 am PST) by brians

There is a rather large mail queue on one of the mail machines in your server cluster. As a result incoming messages are being delayed. Again this is only affecting users hosted in the swarthy cluster. You can access our Panel and click the ” Account Status ” tab to verify the mailserver cluster that your account is hosted on. Our admins are currently investigating the matter. Everything will be resolved shortly. We sincerely apologize for the inconvenience this has caused and appreciate your patience. Please check back for future updates.

UPDATE: Issue has been resolved. All queued mail has been processed. If you hare still experiencing problems, please contact support.


11:14 am

Private MySQL Host Problems

Posted (February 11th, 2009 at 11:14 am PST) by Kyle

One of our Private MySQL host servers is having hardware problems this morning, we have a admin in route to move the machine to new hardware. Machines affected:

psmysql3621  psmysql3623  psmysql3833  psmysql4176  psmysql4223  psmysql4248  psmysql4285  psmysql4294  psmysql4299  psmysql4307  psmysql4603  psmysql5209  psmysql5229  psmysql5720  psmysql5954  psmysql6085  psmysql6099  psmysql6312  psmysql6314  psmysql6356  psmysql6603  psmysql6948  psmysql6994  psmysql7025  psmysql7028 psmysql7170

Update 13:41 PST: After a drive to our other data center, a drive swap and a bit of work to get the machine happy with the new hardware the host server is back up and fully functional.  If you are still experiencing problems please contact our support team from the webpanel.


4:28 am

Morris raid troubles

Posted (February 11th, 2009 at 4:28 am PST) by Sandon
4:25 pm

WordPress 2.7.1 Available!

Posted (February 10th, 2009 at 4:25 pm PST) by Jason C

For those of you who are chomping at the bit to be as cutting edge as possible, WordPress 2.7.1 is now available in our advanced one-click installer. This release is full of bug fixes — so there’s no threat if you don’t upgrade your sites from 2.7 to 2.7.1 right this minute — but staying on top of new releases is always strongly suggested.


3:48 pm

Pan fried

Posted (February 9th, 2009 at 3:48 pm PST) by Leon

Looks like webserver pan has blown a gasket and needs to be moved to new hardware. I’m working on this now and it should be done in about an hour.

Update 4:43pm: All set! If you’re still having a problem with pan, please contact a dreamhost support robot!


 
© 1996-2009, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
5:25 pm

Spunky cluster having some network issues

Posted (February 6th, 2009 at 5:25 pm PST) by Ralph

We are currently working on a problem with the spunky cluster’s network. There is a problem with one of the distribution switches. We have our administration team actively working on this issue right now. ETA for further information is around 20 minutes. We should have a better idea of the problem and solution by then. Sorry about the inconvenience.

Update: This issue has been resolved however we may need to reboot some servers as they may have problems with their network after the switch issue. One of our primary dist switches went down and immediately came back up. This caused a network blip that may require additional work to be resolved. As soon as we feel the issue is completely resolved we’ll post here. ETA about 20 minutes.

Update: This issue should be completely resolved now. If you do still have problems please contact support and we’ll be happy to help.


1:32 pm

Some machines changing IPs

Posted (February 6th, 2009 at 1:32 pm PST) by jeremy

During a recent audit, it came to our attention that 3 servers (skinner, millhouse, and wiggum) had been allocated IP addresses from the wrong pool. In order to fix the issue, we’ve had to re-assign IPs to every domain on these servers. Everything is back up and running and there should have only been a brief period where some sites may have been unreachable while DNS propagated. Everything’s back up and should be noticeably faster than before now that these machines are set up correctly.


3:23 pm

Spacey filer problems!

Posted (February 4th, 2009 at 3:23 pm PST) by Leon

It appears that one of our network filers is having some problems. Admins have been dispatched to the datacenter to check out the problems. This will effect customer websites on the following machines:

appa arizona astore bambi barqs brisk cactuscooler caprisun chai chunky coffee crush crystallight culligan dietrite dominion doubleosting downside espresso fanta fiji flipside frappe fresca fuze gingerale grapico hansens hiball horchata ibc inko koolaid ksharpie ladyanne lipton martinellis minutemaid mountaindew nehi nesbitt odwalla oj orangebang ovaltine perrier pinoysoft pom powerade propel ps5035 rakofi refresco right riptide rootbeer samhall schweppes shasta smoothie sobe sonsuztasarim sprite stewarts tampico tea tropicana trs voss welchs yoohoo zoot

Should be resolved shortly. You can check this by going to the panel > Account Status > Your Email Server. If it’s listed as “Spacey” this may be affecting you.

We will keep you updated!

Update 5:42PM: The issue with the filer is now resolved and everything should be up and running now. If you’re still experiencing issues, please contact our support team!


3:19 pm

giancana server problems

Posted (February 3rd, 2009 at 3:19 pm PST) by danwy

If you’re hosted on the giancana server you may have noticed some downtime currently. It looks like the server may have ran out of memory, but after it was rebooted it had problems loading up one of the startup config files. One of our admins is currently working on this server and hopes to have it up ASAP. Please keep an eye on this post, as we will update it as soon as we have any new information regarding it. We’re sorry for any inconvenience this causes you.

Update – 4:15pm: This should be all fixed up as of about a half hour or so ago. If you’re on this server and continue to have problems, please let our support team know.


8:42 am

Caesar “Et tu, Brute?”

Posted (February 2nd, 2009 at 8:42 am PST) by Kyle

Caesar is having difficulty coming back from a reboot this morning so we are moving it to fresh hardware.  Estimated downtime is 30 minutes.

Update 9:26 PST: It took about 10 minutes longer than expected, sorry about that!


1:15 pm

Dreamhost getting DoS’d

Posted (January 29th, 2009 at 1:15 pm PST) by danwy

Some of you may have been noticing connection issues with your panel and other central DreamHost servers. This is due to DreamHost.com getting attacked by a DoS attack. Our admins are aware of these issues and are working as hard as they can to get things under control. We’ll update this status post once we have things under control. We’re sorry for any inconvenience this causes you.

Update – 02.01.2009 12:05 PM PST: *sigh* This should have been marked as resolved days ago — and we’re terribly sorry for any confusion keeping this post unresolved might have caused. The Denial of Service attack was being mounted by several sites running out-of-date software (including some customer sites) and we managed to quell the attack by the early evening on Thursday.

As was mentioned before, this issue was only affecting DreamHost.com and central services. Your personal sites should not have been seeing any issues during this time. If they were, it was unrelated to the DoS attack that was plaguing us on Thursday. Please contact support if you manage to see any further issues and we’ll gladly do what we can to help sort you out.

Update – 02.01.2009 01:05 PM PST: Apparently giving the delayed all-clear has angered the DoS gods again. The panel is currently slow and the main DreamHost site appears to be down as well. Our Sunday admin is working on clearing this up now and we’ll be sure to keep this entry better updated than it was over the weekend.

Update – 02.01.2009 07:45 PM PST: Okay, the panel and main site are back to normal now. Hopefully this is resolved for good — but if any weird or out of the ordinary things manage to happen, we’ll make a new post here.


8:54 pm

Eastman having issues

Posted (January 28th, 2009 at 8:54 pm PST) by micah

The web server, eastman, is having a few issues tonight. A tech is en route to the datacenter now to take a look at it. It should be back to normal shortly.

Update 1/28 10:55pm: It looked like an issue with memory, but is still having issues. Eastman will be moving to new hardware. Since it seems to be at least up and running for the moment, the hardware changeover will happen at 1am.

Update 1/29 2:15am: The server was moved to new hardware and appears to be working properly.


8:38 pm

Fay acting funk-ay

Posted (January 27th, 2009 at 8:38 pm PST) by Brian@dreamhost

One of our servers, Fay seems to be having some issues with it’s apache web servers. At the moment, we’re still not sure what exactly is causing the problems, but should have more info on it soon! We apologize for any troubles this causes, and will post more info as it comes in!

Update 09:10 pm PST: Ok, this should be fixed now! Seems there was a bad kernel causing all kinds of odd issues. If you still see problems, please report them to our tech support system immediately! =)


4:26 pm

Unexpected Panel Downtime

Posted (January 27th, 2009 at 4:26 pm PST) by Jason C

Due to issues with one of our central databases, our admin team has had to take the web panel down for a brief amount of time while some settings are tweaked. Hopefully the outage shouldn’t last too long — but we’ll keep you posted here as things progress.

Update: One of the databases was caught in a loop due to a programming error and ended up slowing things (such as the web panel) down.  Sorry about that!


5:46 am

SMTP authentication errors

Posted (January 27th, 2009 at 5:46 am PST) by Sandon

It looks like there is cluster-wide issue affecting SMTP authentication. This is causing the inability to send mail. Our admin team is working on the issue and will keep you updated via this status post.

Update: 06:07 AM All service is restored. Please contact support if you are still having any issues.


 
© 1996-2009, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
10:58 am

Quick Frisky downtime 01/26/09

Posted (January 26th, 2009 at 10:58 am PST) by DreamHost

Tomorrow morning, January 26, 2009, at 10:30am Pacific time, we need to do some quick file server maintenance. The downtime is not expected to exceed 30 minutes, with the file server being down for no longer than 15 minutes.
If your “Mail Cluster” shows as Frisky you may be affected by this maintenance, however this file server only serves a small portion of that cluster. Email, web, and ssh/ftp services will be affected, MySQL will be up.

Update! The maintenance is complete and went off without a hitch.  If you are still having trouble with something contact support!


1:13 pm

Eastman having problems

Posted (January 25th, 2009 at 1:13 pm PST) by jordan

We are currently experiencing problems with the shared hosting server “eastman” . The cause of the problem has not yet been identified but we will add to this post as soon as we find out.

UPDATE 2:45 PDT ** The problems with eastman have been resolved. After it crashed it had some problems booting back up again which was what caused the majority of the downtime. All services on eastman should be back to normal.


7:33 am

Otto Blotto

Posted (January 25th, 2009 at 7:33 am PST) by Leon

It seems that the NIC on otto stopped working sometime this morning. I’m currently moving it over to a new machine. Should be back in about an hour. Sorry about that!

Update 8:41: Back up and running. If you are still having trouble, please contact your most favorite support person for any questions!


9:46 am

Nero Falls

Posted (January 22nd, 2009 at 9:46 am PST) by Kyle

The shared hosting server Nero has decided to take the path of it’s namesake and has killed itself. The migration to new hardware is in progress, estimated downtime is ~30 minutes.

Update 10:05 PST: Nero is back up and running on new hardware and services are slowly returning to normal, if you are still having trouble after 10 minutes or so please contact support.


8:21 am

Pukka Getting New Battery Backup Unit

Posted (January 22nd, 2009 at 8:21 am PST) by Kyle

The shared MySQL server Pukka has reported that the battery backup unit for the systems RAID controller is failing. The data on the server is safe without it but since we have a warm body in the datacenter where the server resides we will be preforming the replacement this morning. Estimated downtime is ~10-15 minutes, we apologize in advance for any troubles.

Update 8:30 PST: The BBU was replaced without a hitch and pukka is back doing it’s thing (9 minutes)!


4:16 pm

Spacey networking on the fritz!

Posted (January 19th, 2009 at 4:16 pm PST) by mir

Customers on our spacey cluster are currently experiencing downtime due to networking issues but our admin team is on the case and should have it resolved shortly. We’ll post with more details as soon as they become available.

Update – 5:22 PM, PST Just so you’re not totally in the dark while our admin team sorts this out, here’s a list of servers that may be affected by the ongoing networking issue on the spacey cluster:

7up abrahydroplanes acquire adhoc adverstore aef aims albasoun algrimur ali alignak alpine amp andrewslaterpresents ansible apolo appa appletini aquafina aremes arif arizona artica artofmanliness ass astore ballistic barqs basketmen bawls beebiz bigbrain bigserver binamedia blogdu bluthton bnsf boba bpsites brainsquirt brisk cactuscooler cafecompao calpico calvin cappuccino caprisun carlosrobles cedar cehennem cfinc chai charlotterealestate chiller1 chimneyrock chunky cihan clamato codered coffee coke copping cre8tivegroup crush crystallight culligan cupula daddy dads danjacobs darkmirage dasani davros dbe decoratework delphae dietrite digitmidia divefilmhd django dominion doplr doubleosting downtown dreamer dreamserver drpepper dtvforum dymweb dynweb ecodoc eggnog elroy empire erremedia espresso etxwww evian excel eyedock ezekiel fabsilv fanboy fanta fastlane ffxiii fiji filter flawless flibble fpccprod frances francis-piragua frappe freerick freewebsites fresca frigate fuze fuzzyraygun g3 galford gannon gatas gatorade geyser gingerale glub gluttony goodcow gordini granangular grapico grassroots gravatt grosbouddha hansens happy2008 hi-c hiball hoops horchata hpalumni hstorm hutchtest ibc icculus idmo igordon ims inko intenselighting isabellaval izze izzy jimobi jolt judi keeeeez kennedydesign kestrel kitchensink knutsandvik koolaid ksharpie kvailas l0ki ladyanne lareina leanmean lensman lerftacular levkov lift lingolook lion lipton living8j longcat loronix lusin mab macnimble madrid magictv mamba mandarina marmol martinellis mastequila maxie mccreery mega mella merca milk million minutemaid mirko mmgllc monster mountaindew mrpibb mug musclemilk mycoffee mystic n00bz nastia navi nb nehi neptun nesbitt nestea netant netsolution nhym niagara nimbus nolaflash nuh odwalla oj orangebang ovaltine pacifico painame panda pandion pdxpeace pegproductions pellegrino pepsi permie perrier perrin pickles pig pinoysoft pixy piyonet plainprograms planetapera platext playitserver plentymag pocapower pollos polnetwork pom powerade ppps propel protium ps1 ps3673 ps3711 ps3772 ps3853 ps3902 ps3920 ps3948 ps4088 ps4158 ps4162 ps4185 ps4200 ps4316 ps4348 ps4351 ps4352 ps4353 ps4407 ps4452 ps4578 ps4593 ps4728 ps4800 ps4901 ps4981 ps4984 ps4985 ps4986 ps4987 ps4988 ps4989 ps5035 ps5038 ps5041 ps5119 ps5127 ps5130 ps5147 ps5176 ps5277 ps5448 ps5597 ps5661 ps5701 ps5861 ps5870 ps5902 ps5965 ps6117 ps6145 ps6247 ps6301 ps6332 ps6334 ps6346 ps6369 ps6371 ps6435 ps6443 ps6454 ps6468 ps6469 ps6497 ps6515 ps6537 ps6540 ps6560 ps6691 ps6721 ps6770 pssunkist punch qikserv001 quagmire quixotic rakofi rameniac raoul rawkins rdp redbull redeemer refresco reserver rhetorical rightstuff riptide robot1010 rocketwood rockstar rocky rojax rootbeer rufio rural sai samhall scarface scary schweppes seagull seawolf seltzer sergnet servebooz servidorgva seventeendots sharktopus sharpie shasta silk simpledollar siriusb skeenshare skif skywalker slice slim smaugzone smoothie snapple snidely sobe sodapop solarvistas sonic sonsuztasarim sparkletts sprawsm sprite squeebee squirt stacyfayehan stewarts store storyhole sunkist sunnyd superblog supergate superman tab tallapolitica tampico tang tatiana tea technomatt tejava the-machine threesixty tianjin titania tizer toastyfrog tonic tony trashmenagerie tripter tropicana trs trux uiopluss unadvised uncletapa unipar unosaficionados unusedvps262 urbart usher v8 vanguardps vejaisso voss webapp webcoot webjts webomega webon welchs wellsville wesleybrasil widerfunnel willing wishserver wqli78 xecova xp yoohoo yudomagic zakidesign

We’re still doing whatever it takes to get things back up quickly.

Update – 6:15 PM, PST The admins working on this issue have tracked down the problem to a crafty Denial of Service (DoS) attack and have blocked it. Fortunately, all of this crazy traffic didn’t seem to affect the network file servers very much — so the servers are stabilizing now and things should return to normal in a matter of minutes.


3:55 pm

v8 Moving to New Hardware

Posted (January 19th, 2009 at 3:55 pm PST) by Leon

It appears as though the NIC card for the v8 web server decided to fail! We are currently moving it to new hardware and it should be back up shortly. We’ll post updates as they come!

Update 10:00 PM: We moved it back to the original server… Turns out the NIC was fine but it was grinding to a halt because of the network issue posted above.


5:14 pm

Vex Moving to New Hardware

Posted (January 18th, 2009 at 5:14 pm PST) by DreamHost

Users on the server vex are seeing some downtime right now. This is due to some issues with vex’s hardware. One of our senior techs is working on failing the server over as we speak. Once this operation is complete, we’ll update the post with any and all pertinent information.


8:37 pm

Hera Divorcing Old Hardware

Posted (January 17th, 2009 at 8:37 pm PST) by Kyle

Hera is having trouble with her old body and it’s time to move her to new hardware. Estimated downtime is 30 minutes.

Update: Hera is happily on new hardware, if you are still having trouble please contact support.


10:38 am

giancana and capone moving to new hardware

Posted (January 16th, 2009 at 10:38 am PST) by kitchen

The servers giancana and capone are on our old new platform of raid6 machines. We were unhappy with the performance the raid6 machines provided, so we’re moving them over to raid10 machines, where we are happy with the performance. All of your data will move along with you, in fact, most of it has already been moved. There should be no discernable downtime for anyone except the most brand new users (within the past 20 minutes or so) but we are posting up here anyways, in the event that there is.

We will post another update as soon as this process is complete, which should be within the next 20 or 30 minutes.

Update: 5:46pm Pacific: ok, so the giancana move didn’t go quite so smoothly as we had expected, so we rolled back to the old server temporarily while we figured out the problem. It’s now been moved over to its new hardware for about an hour, and is looking good. Capone has also been moved and should be functioning normally.

If you are still experiencing any problems, please contact support!


 
© 1996-2009, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
3:57 pm

Outdated WordPress Reminder

Posted (January 15th, 2009 at 3:57 pm PST) by Karl

Very shortly we will begin sending out emails to customers with outdated WordPress installations, asking them to upgrade to the most recent available: 2.7. Versions older than 2.6.5 are known to have vulnerabilities that can be exploited for many purposes including defacing your websites, writing arbitrary data to the filesystem under your user (including other domains without WordPress), sending spam, running phishing scams, and various other prohibited or illegal activity. If you are running an older version of WordPress, please upgrade as soon as possible.

If you installed WordPress via DreamHost’s One-Click feature, you can upgrade from the control panel under “Goodies” -> “One-Click Installs” -> “Upgrade or remove previously installed software – Advanced mode”.
https://panel.dreamhost.com/index.cgi?tree=goodies.installer

If you installed manually, you’ll need to upgrade by downloading the most recent version from wordpress.org, here:
http://wordpress.org/download/
They have information available on upgrading prior versions, here:
http://codex.wordpress.org/Upgrading_WordPress

You can keep abreast of future WordPress developments (including security patches) by signing up for their RSS feed:
feed://wordpress.org/development/feed/

The following site can also help you keep informed of known security issues in WordPress or other popular web softwares:
http://secunia.com/advisories/search/?search=wordpress

At this time we are only sending out emails asking people to upgrade, but we may eventually require the upgrades for the domains to remain online: so please upgrade today! Make sure that if you are running any other web software, that it is up-to-date as well. Many popular software packages (including Joomla!, phpBB, Gallery, and others) have known exploits in older versions.

After you’ve upgraded, please also consider installing a spam-screening plugin to ensure your database stays clean and manageable. We have some information about this on the Support Wiki:
http://wiki.dreamhost.com/Fine_Tuning_Your_WordPress_Install#Anti-Spam_Plugins


2:52 pm

Popinski TKO’d

Posted (January 15th, 2009 at 2:52 pm PST) by Leon

Unfortunately, on top of all today’s mania, popinski started goofing up too. We thought it was related to the other issues in that datacenter and missed it earlier. Very sorry about that! It should be back up in about one hour.

Update 4:40: Popinski is back to normal. Should you have any lingering issues, please contact your Happy Dreamhost Technical Support person!


11:50 am

LAX Datacenter Problems

Posted (January 15th, 2009 at 11:50 am PST) by danwy

We’re currently having some issues at one of our datacenters. It is resulting in web and mail downtime for some of our customers on the blingy and frisky clusters. We don’t know what the cause of it is at this point, but our admins are working on the issue and hope to have it resolved ASAP. Please check back here for updates, as we’ll update this post as soon as we have any additional information. We’re sorry for any inconvenience this causes you. Here is a list of possibly affected servers:

athena barracuda belair boxster bugatti cadillac camaro carlititos carrera cayman chevelle continental corvette cutlass dart delorean edsel elcamino eldorado elise enzo esprit fairlane ferrari fiat fiero firebird galaxie gmais gremlin highground hornet hudson hummer hupmobile impala jaguar jetta lancer lesabre malibu manmeat mclaren merkur miata model-t montecarlo murcielago nova olds pacer packard passat phaeton pierce pinto probe prowler rambler rawstock renault riviera rolls scion shelby skoda skylark stingray stratus studebaker tempest tesla thunderbird titan touareg towncar trans-am veyron wrangler

agnus altair birdo bonk boo bowser clank cloud conker daxter dct ecco ermac flamenco fox ganon glassjoe goomba jackie jak jax kirby koopa kyrie lara link luigi mario marth megaman micaiah mordecai nariko nero ness nukem peach pikachu pit popinski ratchet scorpion shinobi slippy snake sothe spyro starfox strider subzero tails toadstool turok voldo yoshi zack zelda zihark

Update: 1/15/2009 – 1:00pm PST: We’ve tracked this issue down to a bad switch in the datacenter which handles most of the storage for these two clusters. Our admins have rebooted it and it is back up and running now. There still may be some issues with individual servers, which our admins are currently working on clearing up now. Everything should be back to normal shortly and we’ll update this post when we believe all the issues have been addressed.


7:20 am

Server Maintenance on 01/16

Posted (January 15th, 2009 at 7:20 am PST) by Sandon

We will be performing some short maitenance on the following 4 servers tomorrow (01/15) at approximately 1 AM PST.

moran, greene, hoffa, dio

Total downtime should be between 5-15 minutes. This status post will be updated when the maintenance is started and when it is completed.

EDIT: 01:00 AM PST: We are starting the maintenance now and the servers will be going down shortly.

EDIT: 01:12 AM PST: All servers are back up. Please contact support if you are on any of the above servers and still having issues.


12:52 am

Good moving to new hardware

Posted (January 13th, 2009 at 12:52 am PST) by DreamHost

Some ram went bad, so it’s gotta move to new hardware.

Update Jan. 13, 2:10pm: It was moved to new hardware and now appears to be running fine.


11:49 pm

Zagnut moving to new hardware

Posted (January 12th, 2009 at 11:49 pm PST) by DreamHost

Unfortunately there is a networking problem on zagnut which appears to be the network card. It’s moving over to new hardware to fix that issue.

Update Jan. 13, 12:24: Found the cause of the issue before the hardware actually needed to be swapped, zagnut should be running fine now.


5:54 pm

Emergency power circuit change

Posted (January 12th, 2009 at 5:54 pm PST) by DreamHost

One of our datacenters informed us today that they have to perform an emergency changeover on a handful of our power circuits. This will result in a reboot of the following servers:

adonis agamemnon ajax aphrodite apollo aries atlas baachus barefoottours calypso charon charybdis chronos coolblue cupid daedalus dewey dione dionysus electra erebus erida eros ganesha gorgon hecate hector helios hephaestus heracles huey hypnos jezebel limbo-randy marbles midas morpheus nanook tartarus vulcan

This will affect web, some mysql, jabber, ftp/shell and email may be slower to check. All of your files and information should be fine. The cutover is expected to take no longer than 30 minutes, and will begin at 10pm Pacific time tomorrow, January 13, 2009.

We apologize for the short notice, we just received word of this ourselves.

Edit 10:13pm: All the circuits are moved, and the servers are back on. We will be monitoring them for problems, but do not anticipate much in the way of further downtime once they boot up. I will post an “all clear” shortly, with any luck!

Edit: 11:14pm: A couple of servers seem to have died on us. janky-vsql01 has now been replaced and we are working on the others.

Edit: 11:46pm: I think we have all of the problems under control. One of the switches lost its configuration and we didn’t pick up on it immediately. I believe all of the servers are up now, please contact support if you are still having problems!


1:04 pm

Cupid is lovestruck

Posted (January 12th, 2009 at 1:04 pm PST) by danwy

If you’re on the cupid server, it’s likely that you’ve noticed the recent intermittent downtime issues with it. We’re currently in the process of setting up a new machine for the server and we believe that it will help alleviate some of the downtime problems it has been having. We don’t have a definite ETA on when this will happen, but the new hardware is in the process of being prepped and we hope to have it moved over today. If you are affected by this server (you can check the ‘manage account’ section of your web panel to see if you’re on this server) please keep an eye on this post, as we will be updating it with information about the hardware upgrade. We apologize for the the inconvenience this has caused you, but we are doing all that we can to remedy these issues.


1:24 pm

Micaiah getting new hardware

Posted (January 9th, 2009 at 1:24 pm PST) by kitchen

We believe the network interface card in micaiah is dying, so we’re going to migrate it over to new hardware. This process will take about 30 minutes and we’ll post an update here once everything’s completed!

Update: 2:12PM PST: the migration process has completed. If you are still having problems, please contact support!


2:30 pm

Gettin’ down to the nitti gritty.

Posted (January 8th, 2009 at 2:30 pm PST) by Leon

Nitti has become overloaded. We’re currently working on getting it balanced and should have it back to normal by the end of the day.


 
© 1996-2009, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
8:13 am

File-Server Maintenance on 01-07

Posted (January 6th, 2009 at 8:13 am PST) by Sandon

We will be taking the following file-servers down shortly for maintenance tomorrow 01-07 at approximately 1 AM PST. If one of the following file-servers is listed when clicking “Account Status” when logged into the panel this will affect you:

lewzealand, julius, wanda

It also might cause slightly elevated load during the down time on the following servers:

achilles, aero, alameda, alondra, archibald, arrow, atlantic, avalon, aviation, babyruth, barry, bazooka, beaudry, bellflower, blowpop, bonbon, bottlecap, boyd, broadway, burton, butler, catalina, centinela, century, cerritos, chex, cognac, compton, cookie, cordial, cupcake, dingdong, doheny, dorito, drumstick, dumdum, edendale, euclid, fairfax, fletcher, foothill, fundip, garth, garvey, gobstopper, goober, good, gower, grand, grappa, gumdrop, haven, heath, hillhurst, hillmont, hoover, hope, hops, hyperion, jefferson, jurupa, lankershim, larchmont, laurel, lemonhead, limbo-spunky4, linq, locust, look, lucas, ludlow, malt, manchester, mayo, melrose, mento, milliken, mills, monroe, nilla, nix, oakdale, ocean, olive, ord, pico, pikespeak, planters, plenty, pocky, poprock, pretzel, price, pringle, razzle, redondo, ritz, rockyroad, rodeo, rum, sake, saltine, schnapps, selma, skittle, snocap, soto, spree, starburst, sticky, sunset, taffy, tampa, temple, tequila, topanga, vanowen, vermouth, warhead, washington, western, whittier, whopper, wilshire, york

The downtime for each file-server should be in the range of 20-40 minutes during the maintenance.

Update 1/7 1:28am PST: We’re about to start this.

Update 4:15 AM PST: Maintenance is complete on all 3 file servers and all services are restored. Please contact support if you are still having any issues.


4:27 am

Aquaman has drowned

Posted (January 6th, 2009 at 4:27 am PST) by Sandon

It looks like Aquaman is having some problems with its raid array and is currently down. Our admins are working on the issue and more information is to come when it is known.

Update 04:51 AM PST: Aquaman is currently being moved over to completely new hardware. We currently trying to copy what we can from the old machine and restore whatever else from backups. Unfortunately this machine is very new and we don’t have many backups available.

More updates to come soon.

Update 05:20 AM PST: So far about 8 of the ~200 GB of data has been copied to the new server. So far it has not had any errors so things are looking good. At this rate we are looking at around 5-6 hours to transfer the rest of the data assuming everything goes well. More updates to come later.

Update 08:00 AM PST: About 50% of the data has been copied so far.

Update: 6:00 PM PST: All services are back online on the aquaman server and have been up for some time now. We experienced no loss in data as a result of this incident. If you are still having problems, please contact support and we will take a look!


12:10 pm

Hooray! Another Rails upgrade!

Posted (January 5th, 2009 at 12:10 pm PST) by Terri

Now that everyone is back to the salt mines for the new year, it’s time for a Ruby on Rails upgrade! 2.2.2 has been released, and we’re not seeing any major complaints about it, so it’s time to upgrade the servers!

We always give our users fair warning because upgrading Ruby on Rails always tends to break currently running applications – but everyone can avoid problems if they freeze their rails! Freezing is way easy. All you have to do is login via to your shell account, change directories into your app, and run:

rake rails:freeze:gems

I’ll be building the packages today and testing them, and if they’re not horrible, I’ll be deploying them across our network slowly as the week progresses. Everything should be done by the end of next week at the very latest!


4:39 pm

Aquaman is sleeping with the fishes

Posted (January 4th, 2009 at 4:39 pm PST) by DreamHost

It would appear that aquaman is currently having issues with serving up some content. Customers have been reporting issues with not being able to see JPG, PNG, GIF, CSS and JS files, amongst others. This means that sites on aquaman will appear to be missing images and theming for the time being — but two members of our admin team are working on fixing this. Updates will come when there’s news of what’s going on.

Update – 6:20 PM, PST: In order to help expedite the resolution of this matter, the server will be inaccessible for the next couple hours. We understand that having a site where people can see some of your content is more desirable than not being able to see anything at all — but I’ve been assured that this will help resolve most issues that customers are seeing.

Update – 7:40 PM, PST: I’ve just received word from the admin in charge of this that all files on aquaman should be served correctly now. If further problems persist, please let our support staff know and we’ll do what we can to make sure you are taken care of.


12:41 pm

DreamHost and Panel Sluggish

Posted (January 3rd, 2009 at 12:41 pm PST) by Leon

We are currently experiencing some slowness on the main DreamHost.com site as well as the web panel (where you get things done!) but are currently working on restoring the services to their full and working order! We’ll post updates as they come!

Update 1:33 PM PDT: The panel and DreamHost site should be back up but we’re still experiencing issues with the sign up pages. More to come…

Update 5:25 PM PDT: We’re still seeing a lot of traffic going toward our signup page. This in turn is still causing the web panel, and some other features (like announcement list signups, and signups themselves!) to have spotty access. We’re doing everything we can to alleviate this, and apologize for the problems! We’ll keep you updated!


5:11 pm

Ruby 1.8.7 Upgrades

Posted (January 2nd, 2009 at 5:11 pm PST) by Dallas

We have started the process of upgrading ruby from 1.8.5 to 1.8.7. This new version is required by some recent ruby software and has been requested several times. In our testing we didn’t discover any issues, but we have now heard about problems with some specific software applications such as RadiantCMS. If you are suddenly having problems with a ruby-based website it may be due to this upgrade and you should check for any updates to the software you are using.


2:51 pm

Legs server being moved to new hardware.

Posted (December 19th, 2008 at 2:51 pm PST) by mir

In light of the recent issues with this server the admin team has decided it would be best to rebuild it on new hardware. There was a problem with the machine (but not the drives) so it either had ram or bad raid controller and rather than have our customers suffer while we pin it down we’re getting everyone onto a working server. This should be completed within the next hour.

Update: Legs back up!
My apologies for the slow update but it appears that the new improved legs has been up for the last 45 minutes now and should provide greatly improved stability!


1:26 pm

Frisky issues (Sorry but no cute title)

Posted (December 19th, 2008 at 1:26 pm PST) by Javier

We had an networking issues with the file server (slapper) connection within this cluster, this has been corrected but unfortunately caused some issues with apache services. Our admins have also corrected this and any sites that had experienced downtime due to this should now have their sites coming up normally, if your are still having issues please contact support.


1:50 pm

yoshi having issues again

Posted (December 18th, 2008 at 1:50 pm PST) by danwy

We thought that the problems with yoshi were through, but it started acting up again earlier today and required a couple of reboots. After the last reboot it didn’t come back up at all. We moved it to new hardware twice now, so we’ve ruled out the possibility of it being a hardware issue. Our admins are in the datacenter and are working on getting this server back up as soon as they can. Please check back here for updates, as we’ll post them as sson as we have them. We’re sorry for the continued issues with this server, but hopefully we’ll be able to find the cause now.


10:44 am

Rack Consolidation Moves

Posted (December 18th, 2008 at 10:44 am PST) by Kyle

We are consolidating several racks to free up space for new hardware (yay!), consequentially there will be some downtime for the servers involved (boo!). Here’s the list of servers that will be affected:

psmysql1871 psmysql1881 psmysql2460 psmysql2461 psmysql2623 psmysql2624 psmysql2648 psmysql2653 psmysql2660 psmysql2839 psmysql2861 psmysql2865 psmysql2882 psmysql2905 psmysql2934 psmysql3050 psmysql3053 psmysql3054 psmysql3131 psmysql3132 psmysql3159 psmysql3164 psmysql3173 psmysql3226 psmysql3229 psmysql3509 hank stormie redbull brisk milk nastia blogdu bpsites fabsilv flawless foobar fuzzyraygun ps3711 ps3853 ps5130 pssunkist stacyfayehan unadvised vanguardps webon zakidesign algrimur apolo bethndn cihan danjacobs mega mystic ps4088 ps4192 ps4316 ps5119 ps5277 ps5585 ps5965 qikserv001 seventeendots usher wesleybrasil zinkweb alignak ansible arif astore davros digitmidia freewebsites hutchtest protium rawkins tajvps technomatt trashmenagerie appletini calvin fastlane frigate judi krillsc raoul skif albasoun delphae dymweb grafia isnmweb keeeeez sonsuztasarim vejaisso brainsquirt carlosrobles cre8tivegroup filter gannon gravatt igordon izzy lorddjinn nhym pacifico ps5597 ps5902 quagmire reserver sharpie snidely artica binamedia esc ffxiii frances gordini grosbouddha idmo ims intenselighting levkov lion permie pickles sai titania andrewslaterpresents basketmen hpalumni ps3948 ps5861 ps6244 scary webapp erremedia grassroots lerftacular lingolook rocknroll sharktopus skywalker storyhole threesixty goodcow mab pollos ps1 ps4593 ps5661 ps6145 tripter trux webcoot adverstore fanboy loronix nolaflash opiniuj pocapower rightstuff rojax adhoc alpine burning cfinc fishbulb kitchensink maozveg ps4097 ps4158 ps4452 ps4800 quixotic

The majority of the servers listed are Private Servers so there are many of them on a single physical machine. Each physical machine will be down for 5-10 minutes, there are 25 machines in all so we are looking at a window of 2ish/3ish hours. Sorry for any inconveniences!

Update: All machines have survived the move! If you are still experiencing troubles please contact support.


 
© 1996-2008, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
5:44 am

gimli has blown a power supply

Posted (December 18th, 2008 at 5:44 am PST) by Sandon

The web-server named ‘gimli’ has had a PSU failure and is currently being failed over (moved over to new hardware). No customer data is stored on this server so the move will not result in any data loss. The fail-over process typically takes approximately 1 hour. This post will be updated once the fail-over process is completed and all services on gimli are back up.

Update 06:05 AM PST: The fail-over process has been completed and all services should be restored. Please contact support if you are on gimli and having any issues with your sites.


5:02 am

Packetloss on some Blingy-cluster machines

Posted (December 18th, 2008 at 5:02 am PST) by Sandon

It looks like we are seeing some packetloss on some of the machines in the ‘blingy’ cluster. This is not affecting all servers in this cluster and as far the problem has been verified on the following servers:

agnus, andrewmarc, anhlong, aspen, bayk, boom, bushido, clarke, clearelement, conker, darth, dobby, dstat, electricmousetrap, erinkel, flamenco, gimme, globalaction, grace, heradlc, jak, kewka, klungo, kyrie, log2ch, markbenj, marth, micaiah, mordecai, nero, ness, nukunuku, pichu, pikachu, pirate, pit, pmu, polgarth, ps4194, ps4233, ps4417, ps4619, ps4879, ps5121, ps5203, ps5274, ps5526, ps5776, ps6125, ps6150, richmond, ryoohki, shareit, snake, sol, sothe, strider, tavdogweb, voldo, wellbored, xdpa, zihark

Our Network Admin is currently investigating the cause. Please check here for updates.

Update 5:15 AM PST: It looks like there was a problem with routing on one of our distribution switches which was causing the problem. Routing has been moved to one of our core routers and the problem is fixed. Please contact support if you are having any issues and are on on the blingy cluster.


7:12 pm

It’s not easy being Greene

Posted (December 17th, 2008 at 7:12 pm PST) by Jason C

Customers on greene will be seeing issues as the server has crashed and requires trudging into the datacenter to give it a manual reboot. We apologize for any issues this may be causing the users affected by this, but hopefully the server should be back up shortly. More updates to come as the issue progresses.

Update – 8:13 PM PST: Aaaaand… we’re back!  Hopefully things should stable for the time being.  If you’re a user on greene and are still seeing issues, please contact support.


1:14 pm

The trouble with McLean

Posted (December 17th, 2008 at 1:14 pm PST) by Jason C

Some users on mclean have been reporting service issues recently.  We’ve discovered that the server is currently saturating the port that it is using to communicate with the outside world.  Since we can’t have that, our admin team is in the process of upgrading the server so it handles its traffic a bit better.

The current estimate from the point person on this issue is about an hour from the time of this post.  We reserve the right to run over that estimate a bit — but if that happens, I’ll do my best to keep the folks suffering from this issue updated.

Update – 1:33 PM PST: Wow! The upgrade went way faster than was expected and I’ve already been given the all-clear. If you’re on mclean and continue to see issues — other than a few slowdowns due to a higher than average load while the server recovers — please contact support and we’ll do our best to work with you.


7:19 am

Yoshi Licked

Posted (December 15th, 2008 at 7:19 am PST) by Kyle

The webserver yoshi is exhibiting symptoms that hint at impending disk failure, the server is being taken offline and is being moved to new hardware. Estimated downtime is 30 minutes.

Update 8:00 PST: Yoshi is taking a bit longer to get back on it’s feet, will update when hardware migration is complete.

Update 9:40 PST Yoshi is back in business, sorry for the delay!


2:25 pm

Spacey filer problems

Posted (December 14th, 2008 at 2:25 pm PST) by danwy

We are currently experiencing some networking issues with one of the file servers in our ‘spacey’ cluster. If you’re on one of the servers below and are having problems with your website, this should be resolved soon. One of our admins is on his way to the datacenter to work on the problems with this file server and fully restore service.

7up amp aquafina arizona barqs bawls boba brisk cactuscooler calpico cappuccino caprisun chai clamato codered coffee coke crush crystallight culligan dads dasani dietrite drpepper eggnog espresso evian fanta fiji frappe fresca fuze gatorade geyser gingerale grapico hansens hi-c hiball horchata ibc inko izze jolt koolaid lipton martinellis milk minutemaid monster mountaindew mrpibb mug musclemilk nastia nehi nesbitt nestea niagara odwalla oj orangebang ovaltine pellegrino pepsi perrier plentymag pom powerade propel punch redbull refresco riptide rockstar rootbeer schweppes seagull seltzer servidorgva shasta silk simpledollar slice slim smoothie snapple sobe sparkletts sprite squirt stewarts sunkist sunnyd tab tampico tang tea tejava tizer tonic tropicana v8 voss welchs wishserver yoohoo

Update – 12/14/08, 5:19 PM PST: So it turns out the file server is in a bit of a reboot spiral due to a few bad drives. Our admins are in the process of preparing and swapping the bad drives and hopefully things should level out soon. More updates will be posted as soon as things are all clear.

Update – 12/14/08, 11:13 PM PST: As it stands, the file server is going through disk checks to make sure that nothing has been lost and that a bigger problem isn’t going to bite everyone affected by this unfortunate outage later on. This is going to take a few hours to do — but once the file server is happy, we’ll update this entry.

Update – 12/15/08, 3:31 AM PST: I’d like to say that the checks were a complete success — but they weren’t. The operations center technician who is left in charge of this is getting a hold of one of our senior admins and they will be working to resolve this. More as things progress.

Update – 12/15/08, 3:31 AM PST: Unfortunately it was determined that if we could save the filer at this point, it may take longer to do that than to replace the hardware. One of our admins is currently handling that. There isn’t an estimate for restoration of service at the moment, but another update should be coming soon.

Update – 12/15/08, 11:31 AM PST: We are unfortunately still having problems accessing the data on the file server. We are going to try upgrading the Operating System in the hopes that this will allow us to access your data so that we can move forward. We will know if the upgrade is helping in roughly 3 hours, if everything goes well. At that point we will do our best to send another status post detailing the plan at that time.

Update – 12/15/08, 2:15 PM PST: We have isolated affected customers, and users not on the peeler file server are back up and running. Unfortunately, customers on peeler are still experiencing an outage. For those of you on the problem file server we are going to be moving you to a new machine. Unfortunately we are not able to access and move your data, but the move will allow you to upload your locally backed up content and get your email working. We are still attempting to recover data, but are not optimistic at this point. We are in a “wait and see” scenario with the rebuilding of peeler’s array. This process will finish late this evening at the earliest.
If you are on the problem file server we will be contacting you by email as soon as your account is moved with more details and instructions on getting back up as soon as possible.

Update – 12/16/08, 00:30 AM PST: Peeler is back up without data loss. We’ll begin the process of copying the data over to the new servers soon. Thanks again for your patience.


10:27 pm

Tang moving to new hardware

Posted (December 11th, 2008 at 10:27 pm PST) by DreamHost

There was a security issue with the server tang, and to be safe, the server is moving to new hardware.

–Update: The move went smoothly and everything appears to be working now.


11:37 pm

Digdug not digging it

Posted (December 10th, 2008 at 11:37 pm PST) by Terri

Digdug’s boot disk is pushing up daisies. We’re going to replace it as fast as possible, but it will take us a couple of hours to get it back up and running. We apologize for the inconvenience.

Update: 1:42 AM PST. Digdug is now happy on new hardware and all services should be restored. The new hardware is newer/faster so any users on this server should see better performance on the new hardware.


7:50 pm

WordPress 2.7 now available!

Posted (December 10th, 2008 at 7:50 pm PST) by Jason C

There’s a brand new version of WordPress available in the advanced one-click installer — and boy is it pretty.  The WP team has spent a lot of time and put quite a bit in the way of resources towards making things both look and work nicer.  This release doesn’t contain any security fixes that we’re aware of, but it definitely feels quite a bit faster.  Well, at least in our rather unscientific tests it does.  Your mileage may vary, of course…

Give your installs an upgrade when you’ve got some free time.  And if something goes totally haywire, our support team is here and totally happy to help.

P.S. Our web panel team assures us that the easy one-click installer will be upgraded shortly.  Keep an eye on this post for updates.


7:15 pm

Schilling is down

Posted (December 10th, 2008 at 7:15 pm PST) by Terri

Schilling has lost a NIC, so right now we’re transferring everything to new hardware. This shouldn’t take longer than an hour. We apologize for the inconvenience.

UPDATE: Schilling is back up and ticking along. If you’re having any further problems, please contact support.


 
© 1996-2008, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
8:12 am

Spacey File Server Issue

Posted (December 10th, 2008 at 8:12 am PST) by Justin@DH

Our oncall admins watching our servers have noticed an issue with a file server in the spacey cluster. The file server is being examined and worked on. The file server currently being offline will cause some downtime on a small number of apache services on the following servers. We are working to get this resolved as soon as possible. If there is any further information we will update this post.

Server possibly having downtime due to this issue:

amp arizona barqs bawls boba brisk calpico cappuccino caprisun carlosrobles chai clamato codered coffee coke crush crystallight culligan dasani dietrite eggnog esc evian fiji frappe fresca fuze geyser gingerale hansens hiball ibc inko izze jolt koolaid lingolook lipton martinellis minutemaid monster mountaindew mrpibb mug musclemilk nesbitt nestea orangebang ovaltine pellegrino powerade propel refresco riptide rockstar rootbeer schweppes seltzer shasta slice snapple sparkletts squirt stewarts sunkist sunnyd tab tampico tang tea tonic tropicana unosaficionados v8 webjts welchs yoohoo

— UPDATE Wed Dec 10 09:27:27 PST 2008 —

The file server is back in working order, and we are verifying the last of the servers are all using it correctly. If you are still having issues that seem related to the above, please contact support. Sorry again for the downtime.


11:15 am

Panel Maintenance

Posted (December 9th, 2008 at 11:15 am PST) by DreamHost

We will be taking the DreamHost Webpanel offline for maintenance early tomorrow morning. The downtime window will begin at Wednesday, December 10, 1:00am PST and extend until 3:00am PST. Likely downtime within this window is 1 hour.

Update Wed Dec 10 02:35:56 PST 2008: All finished. Panel back online.


7:03 am

MySQL Maintenance on 12/10

Posted (December 5th, 2008 at 7:03 am PST) by Sandon

The following MySQL machines will be going down between 12:30 AM and 2:30 AM PST on 12/10 (Wednesday) for maintenance:

tammany, elmyra, skimbleshanks, tachibana, sarabi, streaky, ragland, jingoro, drooper, tomoe, odie, lassie, scooby, snagglepuss, natty, cujo, cerberus, sekhmet, hooch, sparky, fluffy, bingo, sanspoof, yeller, krypto

The total downtime for each machine should be in the range of 15 minutes for each machine but but the window for the entire set of servers is between 12:30 and 2:30 AM.

Update 11:59 PM PST: Preperations for this maintenance are running a bit behind schedule so this is being pushed off another hour or so. This means machines will be going down around 1:00 – 1:30 AM PST.

Update 1:05 AM PST: Starting maintenance. Servers will start going down now.

Update 3:30 AM PST: All services have been restored and each server was down between 10-20 minutes. None of the servers exceeded 25 minutes in downtime.


7:00 am

Files Forever Service going down temporarily 12/09

Posted (December 5th, 2008 at 7:00 am PST) by Sandon

The Files Forever service we offer will be offline temporarily on Tuesday 12/09 at 12:30 AM PST. The outage should not exceed one hour and should be completed between 30-45 minutes for maintenance.

Update 12:32 AM PST: The service is going down right now.

Update 1:56 AM PST: Things took longer than expected but all services for file-forever should be restored.


7:55 am

Snocap Capped

Posted (December 2nd, 2008 at 7:55 am PST) by Kyle

The shared hosting server snocap is showing some unhealthy signs this morning and is being moved to new hardware. Estimated downtime is 1 hour.

Update 8:39 PST: As it turns out a single site received a huge influx of traffic, enough so that the server would crash minutes after reboot.  Snocap is holding steady on it’s current hardware now that we have taken appropriate action but we will continue to monitor it for good measure.


6:16 am

Garvey No So Gravy

Posted (December 2nd, 2008 at 6:16 am PST) by Kyle

The shared hosting server garvey is acting unpleasant this morning so we have opted to move it to new hardware. Estimated downtime is 30 minutes, sorry!

Update 6:50 PST: Garvey is on new hardware now and configuration updates are finishing up, it’s all gravy now!


6:59 pm

28 Days Later

Posted (December 1st, 2008 at 6:59 pm PST) by Jason C

To cut down on the amount of spam that Akismet has to sort thru the DreamHost Status blog (which you’re reading right now) will stop accepting comments on entries older than 28 days.  You are still totally welcome to comment and commiserate with fellow users on new issues, but we thought it might be nice to give our filters a break as an early christmas present.

To give you an idea of how many spammy comments are submitted to the blog on a daily basis, Akismet has filtered out 111,468 messages for us between the time we’ve implemented it and now.  That might not sound surprising to some of you, but we only started using it as our spam filter in June.

That’s right, June!  Of 2008!  Crazy, right?

So why are we taking away the ability to comment on these older entries?  Well, for starters, once an issue has been resolved, there’s not a lot of chatter surrounding it.  In fact, the conversation dies down pretty quickly.  On top of that, sifting thru the spam queue for this blog shows that most of the junk comments happen on the older entries.  This is a tactic used by spammers in the hopes that the site operator isn’t savvy and won’t pay attention to a bunch of junk piled into an entry from a year or two ago.  And it works pretty well — which is why they keep doing it.

So instead of encouraging poor behavior on the part of the community of blog spammers out there — and that’s kind of what we’ve been doing up until now — we’re going to not even give them the opportunity to be jerks.  I’ll be keeping an eye on this entry, so if you have any constructive criticism on this countermeasure, I’ll do what I can to take it into consideration moving forward.  Thanks!


9:55 pm

Email on our cluster “spunky” is delayed =(

Posted (November 29th, 2008 at 9:55 pm PST) by Brian@dreamhost

Currently, all inbound mail on our “spunky” servers is backed up. We’ve narrowed this down to a problem with the mysql database system that routes mail for this cluster. We’ve restarted the mysql database involved in routing mail, however, it does take sometime to sync up the database slaves.

I should mention that mail should be queued up, and not bounced, so mail should not be lost while this is happening.

Since spunky is our largest cluster, it does take sometime to get it back on it’s feet, however, we’re doing everything we can to get this back up and working correctly. We apologize for any issues this may have caused! Stay tunes to this post for more details as we get them.

UPDATE: The servers have processed the backed up e-mail, and everything is back to normal now. There was a lot of e-mail, and so it took a very long time. We apologize for the inconvenience.


6:39 am

Ritz is Cracked

Posted (November 28th, 2008 at 6:39 am PST) by patrickm

The Ritz server is experiencing hardware failure!  We are in the process of failing this server over to new hardware.  This process should not take too long and we apologize for the inconvenience that this has caused!  We will update this post with more information, just as soon as we can.

Update 8:02am: The failover is complete and we’re just running through some final configs now.  The server is back up now, but if you are still having any issues, please feel free to contact Support and we’ll be happy to check it out!


10:58 pm

Get a brain moran!

Posted (November 26th, 2008 at 10:58 pm PST) by Leon

It looks like moran has some trouble with one of its drives. I’ll be swapping it out tonight and it will be down for about 15-20 minutes. Sorry about that, I’ll make it quick!

Update 11:24pm: I shut it down around 4 minutes ago, swapped the drive and it’s already back up. It’ll be rebuilding its array for a bit, but you shouldn’t notice anything weird. If you do, please send a note to your friendly Dreamhost support person!


 
© 1996-2008, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
5:10 pm

Sander filer swap Wednesday morning

Posted (November 24th, 2008 at 5:10 pm PST) by DreamHost

This wednesday, Nov 26th 2008, shortly after midnight Pacific time we will be swapping the file server “sander” for new hardware. The old hardware is showing correctable memory errors, and changing the memory a few weeks ago did not clear this up. This will only affect you if you are hosted on that file server and are in the mail cluster “randy”.

The estimated total downtime is 30 minutes. During this time your website, email, and ftp/ssh access will not work. MySQL will be unaffected.

Update: 11/25 11:56pm — We’re about to handle this. Sander will go down in about five minutes. The hardware swap should take less than fifteen minutes, with service returning less than fifteen minutes after that. Thanks for your patience.

Update: 11/26 12:20am — The hardware swap was finished about ten minutes in, and services are now returning.


8:27 am

Baseline Benched

Posted (November 21st, 2008 at 8:27 am PST) by Kyle

The web server “baseline” has been having issues with one of it’s network interfaces and consequentially has required numerous reboots in the last few days. Baseline is being moved to new hardware so this won’t be a problem in the future and should be back to normal in about 30 minutes.

Update 10:20 PST: The move to new hardware is going slower than it normally does, we are working on it and hope to resolve it shortly.

Update 14:31 PST: The move finished around noon but forgot to provide a update, sorry about that!


8:05 am

Private Server Host Instability

Posted (November 21st, 2008 at 8:05 am PST) by Kyle

This morning a large number of Private Server hosts started misbehaving around 4:30. The affected servers required a reboot and most of them are already back to doing their thing. We are investigating the cause of this issue to avoid a repeat of this mornings event. If you are still experiencing an outage contact support with as many details as you can muster and accept our apology for any inconveniences.

Update 9:55 AM PST: We are still noticing outages across various PS machines. Our admins are diligently working towards pinpointing the root cause of this issue and implementing measures to prevent future occurrences. Again, we sincerely apologize for the inconvenience this has caused and appreciate your continued patience. Please check back for future updates.

Update 2:15 PM PST: This appears to be related to some issues with our security daemon and recent changes to the Private Servers. To be honest, this was a hard issue to pinpoint — but the tech who has been monitoring our network for the past few hours has assured me that PS’es have stabilized. If you see issues moving forward, please let our Support Team know right away and we’ll gladly do what we can to drill down on whatever problems you might be seeing.


4:38 am

Swarthy-vserver01 hard-drive failure

Posted (November 21st, 2008 at 4:38 am PST) by Sandon

I am sorry to report that the hard-drive on swarthy-vserver01 has failed so I am failing it over to a new set of hardware. For a vserver this typically takes approximately 1 hour to complete. This is only affecting you if you have a PS named any of the following:

alexandraerin, cccs, chippingford, dragonu, ejflavors, freewill, ps5289, sofanaranja, ventures, zalanthas

Update 6:00 AM PST: As of about 15 minutes ago All services on the above PS’s should be restored. Please contact support if you are still having any issues accessing your sites.

–Happy Late-Night DreamHost admin team.


2:06 pm

Dribbler’s been drooling

Posted (November 19th, 2008 at 2:06 pm PST) by Jason C

When it rains, it pours. Another network file server (dribbler) that serves the janky cluster was running into issues that didn’t require any upgrades (unlike the earlier issue with swiffer). As it turns out, Murphy and his blasted law apparently wanted to kick us while we were down. The servers that were seeing problems because of this were:

abraxor adbo adonis agamemnon ajax alboz ampedsports aphrodite apollo arda aries atlas atra avenus avi2l azcuehnos baachus backtrack3 barefoottours bashtage batserver battle bbd ben berniemedia bettyboop bgportal bimbodulcop blackfalcon-web blackfday blackhole blackman blizzard blogshare bluebox bogus borealis breckgear buychina caiman calypso caparazon casaclubtv casper celeb centaurus cfndc charon charybdis chens cheritek chipsiweb chronos chubu ciclo22 concorde coolblue daedalus dalliance danyun designerbrickmarble dexau dh01 dione dionysus diversity djm djpreet dusqi dusty ecards eduagora electra elvis emanuel entropia erebus erida eros evildead ewok farnsworth fastserver fiercee fifachile flamjam flowstudios forkpile fresh fuzzacademy ganesha gazihost gencer gl gladys gnot gorgon grasshopper grimbald gumoz habbid hades hb02 hdotnet hecate hector helios hephaestus hera heracles hermes hestia hypatia hypnos i2 icarus ichigo iinst1 instantcenter iota iotatec iris itauna jarcas jbutton jelly jlaurinphotography jobbr jove kaysnature kermit kesido keyiflimutfak keysersoze kiley kitwe kopi kratos kronos kztixo lagrange leichtman leto linkmoney linneo lolicg lucent lucy madotter managedq marcdorcel marhgil maserver mediadirect medusa mf1 mf2 mho midas minerva mll mooshi morpheus muchmedia mwl nandobhz nazgul nemesis netseyredin noodledee numbers oceanone odysseus odysseus1 odyssey oedipus optimusprime oriza osiris pan pandora peaceful pegasus persephone phi pichoster pimp plubble pmbd poseidon privateserver prospero proteus proxybilisim ps3854 ps3865 ps3868 ps3908 ps3930 ps3944 ps3953 ps3957 ps3982 ps3994 ps3997 ps4053 ps4072 ps4098 ps4121 ps4131 ps4168 ps4206 ps4337 ps4555 ps4571 ps4582 ps4583 ps4584 ps4585 ps4693 ps4721 ps4729 ps4730 ps4731 ps4978 quifflet quiknet reactor reuben rhea rhinoco rvooz serieonline shyfry simflight sisyphus sitesablaze sixtytwo smutbox socket somaticvision sturman sturmgroup stusserver sunnywood svetluska synergia tailfin tamiyaclubmovies tartarus tcb tethys thanatos thetis tic tonni triage triton twunbbs tyche universoprivate viigmt vivenet vrt7 wally warppipevps webben webhostservice weblogstematicos webserver webxl wolkanca xiondesigns yestoronto yuxiong zdki zentastic zeus

To keep this from happening again, some offloads will be taking place. Those should be pretty low key tho. Most servers should be back and operating correctly now — but a few are being pulled up by the scruff of their neck. If you are on one of these machines and happen to still see issues, please contact support and we’ll gladly help you get your sites back up as quickly as possible.


12:15 pm

Swiffer file server problems

Posted (November 19th, 2008 at 12:15 pm PST) by Ralph

We are currently experiencing some problems with the file server Swiffer. The machine is having problems with some ram that needs to be replaced. We are on the way to replace it right now. The affected machines are:

abrahydroplanes aims arizona artofmanliness asle bambi boba brisk cactuscooler calpico caprisun cedar chai clamato codered dasani dietrite downside eggnog evian eyedock fabsilv fiji flipside frances francis-piragua frappe freewebsites fresca gannon geyser gordini grapico grassroots gravatt horchata inko intenselighting mastequila mccreery minutemaid mrpibb mug musclemilk nehi nesbitt netsolution niagara odr oj orangebang ovaltine polnetwork pom powerade ppps ps1 ps3673 ps4316 refresco right seltzer shasta silk sparkletts stacyfayehan sunnyd tampico tang tea tizer tropicana uncletapa welchs xecova zakidesign zoot

We will post as soon as this issue is resolved.

Update 13:26 PST: The ram has been replaced and the file server came back online without a hitch. We are cleaning up the webservers that tanked due to I/O backup, if your sites are not back now they should be shortly.


10:54 am

capone hardware failures

Posted (November 17th, 2008 at 10:54 am PST) by danwy

The server, capone, is currently having some hardware problems. One of our admins suspects that the Nic card went out and is currently on his way to the datacenter to swap the hardware out. We should have this back up shortly and we’ll update this post once it’s resolved. We’re sorry for the inconvenience this causes you.

Update 12:15pm PST: This server is now back up and running as of about 10 minutes ago. We’ve tested several sites hosted on the server and they’re back up and running, so things should be back to normal now. Of course, if you still have problems please contact support and we’ll be happy to look into it further.


3:00 am

Three MySQL servers going down shortly for maitenance

Posted (November 15th, 2008 at 3:00 am PST) by Sandon

The following MySQL servers will be going down shortly for some brief maintenance. Estimated downtime is between 15-20 minutes:

dewey, huey, janky-vsql01

If you have a PS named any of the following you are on janky-vsql01 and this will affect you:

psmysql1829, psmysql1861, psmysql1863, psmysql3011, psmysql3016, psmysql3029, psmysql3166, psmysql3171, psmysql3228, psmysql4330, psmysql4588, psmysql4975, psmysql744, psmysql745, psmysql797, psmysql799, psmysql802, psmysql961, psmysql962, psmysql963, psmysql964, psmysql965, psmysql966, psmysql967, psmysql968, psmysql969, psmysql970, psmysql972, psmysql975, psmysql978, psmysql979, psmysql986, psmysql991

I apologize for the short notice of downtime.

Update 3:18 AM PST: Ok, dewey and janky-vsql01 are back up so all services should be restored on those. Huey is doing an fsck due to the fact it has been a considerable amount of time since the last one was ran and a check was forced. MySQL machines are fast so I would estimate it will be back up within 5 minutes.

Update 3:24 AM PST: Ok. huey is back up and happy again. All service has been restored on all machines. Please contact support if you are on any of the above machines and still having any issues.


1:11 am

Spunky-vsql01 going down shortly for maitenance

Posted (November 15th, 2008 at 1:11 am PST) by Sandon

This server will be going down in the next 30 minutes or so for some short maitenance. This will effect you if your PS is named any of the following:

psmysql1014, psmysql1475, psmysql1476, psmysql1479, psmysql1686, psmysql1801, psmysql1832, psmysql1869, psmysql1872, psmysql1886, psmysql1887, psmysql2430, psmysql2431, psmysql2433, psmysql2435, psmysql2436, psmysql2543, psmysql2674, psmysql2676, psmysql2677, psmysql2845, psmysql2846, psmysql298, psmysql3172, psmysql3196, psmysql332, psmysql3398, psmysql3404, psmysql3733, psmysql3812, psmysql383, psmysql386, psmysql387, psmysql3875, psmysql389, psmysql3899, psmysql391, psmysql3933, psmysql3935, psmysql3941, psmysql401, psmysql4037, psmysql4136, psmysql4177, psmysql4182, psmysql4244, psmysql428, psmysql4600, psmysql4688, psmysql4689, psmysql4736, psmysql4873, psmysql4972, psmysql746, psmysql800

Total downtime should be approximately 10 minutes and happen in the next 20-30 minutes. I apologize for not giving a more advance notice.

Update 01:44 AM PST: Ok all services should be restored. Total downtime was around 12 minutes.


10:58 am

Panel is sluggish

Posted (November 14th, 2008 at 10:58 am PST) by Terri

A super huge monster query has made our central database choke, and thus the Webpanel is very slow right now. We are working to clear out the query and track down where it is coming from so this does not happen again.

We apologize for the inconvenience. We will post more information as it becomes available.

UPDATE: We’re shutting down Webpanel web services for a little bit to give the database a bit of breathing room while we investigate this issue. This will shut down the Webpanel down completely for awhile.

UPDATE: Our central database is back up and running. We’ve altered the query that caused the problem and so this shouldn’t happen again.


 
© 1996-2008, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
9:06 am

Shemp, shempy.

Posted (November 14th, 2008 at 9:06 am PST) by Kyle

Shemp is one of our older servers and it seems to have met it’s inevitable fate. It is being failed over to new hardware, estimated downtime is 30 minutes.

Update: New shemp is up and rocking a new pair of shorts, apaches configurations are running but should complete shortly


1:04 am

Web Panel DOWN

Posted (November 14th, 2008 at 1:04 am PST) by JeffH

We do apologize for the inconvenience…our Web Panel is currently down, and we have an admin working furiously to restore it as soon as possible.  We don’t expect this will generate much support, seeing as how no one can file a ticket at the moment…but this is a heads up to let you know we are aware of the problem and are fixing it.  Again, sorry for any hassle!

Update 1:44 AM PST The panel should be back up and working again. Thanks for your patience.


6:45 pm

Problems with watanabe (Resolved)

Posted (November 13th, 2008 at 6:45 pm PST) by Jason C

We’re quite sorry to say that users on watanabe will be running into problems due to hardware failure.  A member of our admin team is currently working on resurrecting the server from the dead – but doing so has proven to be a bit problematic.  Please know that we’re doing whatever we can to get the sites of users on this server back up as soon as possible.  Further updates will be forthcoming as this matter develops.

Update – 8:39 PM PST: Good news!  The drive recovered!  However, in the interest of preventing data loss, our admin team has decided to fail the machine over to totally new hardware and copy the files over from the old drive.  This does mean that your data will be unavailable while everything copies over. Since this is a newer server tho, I’ve been assured that it shouldn’t take too long for everything to be migrated.

I know this might be a very big problem for some folks — but we’d rather prevent the server from having to recover a larger amount of files somewhere down the line. Expect more updates and progress reports here shortly.

Update 12:35 PM Pacific Friday Nov. 14 2008 We are still restoring data, we are up to the “t”‘s in the alphabet assuming rsync is going in order. You may see skeleton directories present without your data if you are after “t” in the alphabet, the data from the old server should fill in those gaps!

Update 3:00 PM Pacific Friday Nov 14 2008 The rsync onto the new hardware finished. If you are still missing data please contact technical support and we will see if we have it on the old server. We apologize for the inconvenience of your data being down for a whole day like this.


1:53 am

Dalitz is fscking

Posted (November 13th, 2008 at 1:53 am PST) by Sandon

Dalitz was unstable and had to be rebooted. While it was booting back up a file-system check was forced which is causing a delay in it coming back up. This typically takes 1-2 hours. Please accept my sincerest apologies for this inconvenience and check this space for updates.

Update 4:20 AM PST: The file-system check is about 80% complete.

Update 4:49 AM PST: The file system check is complete and all services are restored. If you are still having any issues connecting to your site please contact support.


9:18 pm

Gotti Got Filesystem Checks

Posted (November 12th, 2008 at 9:18 pm PST) by Patrick

During a routine kernel upgrade, an oddity in debian’s e2fsprogs package forced a filesystem check on gotti. Unfortunately, as has been commonplace on this status page lately, this is going to take an hour or two. Please accept my sincerest apologies for this inconvenience and check this space for updates.

– Update Wed Nov 12 23:56:35 PST 2008 –

Annnnd we’re back! The good news is that this shouldn’t happen again on any future kernel upgrades!


5:49 am

Trafficante is down

Posted (November 12th, 2008 at 5:49 am PST) by Sandon

The server ‘trafficante’ was rebooted and is currently running a file-system check. This typically takes ~1 hour to complete. This status post will be updated once the it is done and the machine finishes booting.

Update 7:21 AM PST- It looks like the file-system check has taken longer than expected. Its still going and is about 80% done.

Update 8:14 AM PST. All services should now be restored. Please contact support if you are still having any issues connecting to the server.


7:21 pm

Burke down for the count!

Posted (November 11th, 2008 at 7:21 pm PST) by Brian@dreamhost

It seems that Burke has crashed today, and needs to be rebooted. The server is coming up now, however, like capone, needs to be checked. This check can take several hours. We’re working on getting this machine up as fast as possible, and apologize for the problems this causes!

– Update –

Burke is back up and serving data. If you are on burke, and are still having problems with your service, please contact support.


5:30 pm

Capone is on its way back up

Posted (November 11th, 2008 at 5:30 pm PST) by andrea

The server Capone was having problems, so while looking into it we soft rebooted it. Unfortunately, it didn’t come back from that reboot, so we had go for a complete reboot (powercycle). The server is now coming back up, but it’s running a file system check to ensure data integrity. It’s a big server, so it’s taking a while, very sorry about that.

Please check back here for updates!

– Update –

Capone has been back up and serving data for an hour or so now. If you continue to have any problems at all, please contact support!


8:17 am

Problems with heath

Posted (November 11th, 2008 at 8:17 am PST) by andrea

Those of you on heath (check Account Status in web panel if you’re not sure if this is related to you) probably noticed by now that the machine was rebooted earlier this morning. It came back fine, but it appears to have crashed again. The admins are moving it to new hardware as it turns out this one has bad NICs. The move shouldn’t take long and it doesn’t affect your data, only your web service (as in, your files are still there, only your site won’t show in a browser until we bring the server back up). We’ll post an update when the process is done.

Please accept our apologies for all inconvenience this has caused!

Update 11/11/2008 9:10am: the server is back up again, and all services are running. We checked some domains to verify, but if you continue having issues with your sites on this machine, don’t hesitate to contact support for assistance.

Update 11/11/2008 12:20pm: It appears that the machine is again having problems. One of the new machine’s network interface is having problems, so it looks like we’re dealing with bad NICs again. We’re keeping an eye on the machine, and will be installing new NICs (network interface cards) later tonight. Sorry about the continued inconvenience. Please bookmark this page and keep checking the status post for updates.


2:56 pm

bonanno server having problems

Posted (November 9th, 2008 at 2:56 pm PST) by danwy

Our server, bonanno, is experiencing some uptime problems. While investigating some issues on it, the server became unresponsive to our admins and required a reboot. Unfortunately, it never came back up after it was rebooted. Our admins are on their way down to the datacenter now in order to get this back up and running. As soon as we have any updates, we’ll update this post with any information we have. We’re sorry for the inconvenience this causes you.

Update – 2:45pm PST: The server should be back up and running now. We’ve tested many customer sites on the server and verified that they are working. Our admins found the reason for it not coming back online after the reboot and have corrected it so that it will not happen again in the future. If you’re hosted on this server and are still having problems, please contact support to look into your issues further. Again, we apologize for any inconvenience this caused you.

p.s. the original time of this should have been 1:56pm, not 2:56pm.


 
© 1996-2008, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
10:21 pm

abrego issues

Posted (November 7th, 2008 at 10:21 pm PST) by kitchen

We are currently experiencing some hardware issues with the abrego server. We are looking into the problem right now and hope to have a resolution soon. We will post updates as things progress.

Update 10:22PM PST: It appears as though the raid controller decided it didn’t like its raid anymore, and dropped it on the floor. Fortunately, everyone on this server was just recently moved to it, so we are moving everyone back to their old machines right now.

Update 1:32AM PST: sites have been restored back to their original locations and should be back up and running now. If you are still having problems, please contact support.

Thank you very much for your patience.


2:27 pm

VSQL Host Maintenance

Posted (November 7th, 2008 at 2:27 pm PST) by Kyle

We are taking down one of our VSQL hosts for mandetory maintenance, the window of downtime should be approx. 20 minutes.  Affected Private MySQLs:

psmysql3838  psmysql4070  psmysql4271  psmysql4402  psmysql4590  psmysql4779
psmysql3849  psmysql4080  psmysql4280  psmysql4403  psmysql4598  psmysql4880
psmysql3864  psmysql4148  psmysql4289  psmysql4404  psmysql4614  psmysql4887
psmysql3873  psmysql4153  psmysql4296  psmysql4408  psmysql4615  psmysql4893
psmysql3884  psmysql4174  psmysql4308  psmysql4428  psmysql4744  psmysql4935
psmysql3891  psmysql4180  psmysql4313  psmysql4429  psmysql4747  psmysql4938
psmysql3895  psmysql4213  psmysql4314  psmysql4437  psmysql4750  psmysql4942
psmysql3907  psmysql4228  psmysql4317  psmysql4439  psmysql4751  psmysql4971
psmysql3967  psmysql4229  psmysql4318  psmysql4441  psmysql4753  psmysql5007
psmysql3974  psmysql4253  psmysql4322  psmysql4556  psmysql4754
psmysql4018  psmysql4254  psmysql4323  psmysql4558  psmysql4756
psmysql4047  psmysql4256  psmysql4395  psmysql4568  psmysql4757
psmysql4051  psmysql4261  psmysql4399  psmysql4569  psmysql4766

This is a one time deal and will not need to be preformed frequently.

Update: All done, MySQL Private Servers are coming back online right now.

Update: Doing another host, same deal.. affected servers:

psmysql3840  psmysql3968  psmysql4332  psmysql4359  psmysql4388  psmysql4489
psmysql3867  psmysql3975  psmysql4333  psmysql4369  psmysql4390  psmysql4605
psmysql3876  psmysql4019  psmysql4334  psmysql4376  psmysql4391  psmysql4609
psmysql3887  psmysql4043  psmysql4335  psmysql4378  psmysql4393  psmysql4610
psmysql3892  psmysql4067  psmysql4354  psmysql4379  psmysql4478  psmysql4611
psmysql3896  psmysql4076  psmysql4355  psmysql4380  psmysql4479  psmysql4612
psmysql3911  psmysql4101  psmysql4356  psmysql4381  psmysql4480  psmysql5004
psmysql3937  psmysql4331  psmysql4358  psmysql4385  psmysql4483

Update: All done!


11:06 am

Scalise Dirtnap

Posted (November 7th, 2008 at 11:06 am PST) by Kyle

Scalise went down unexpectedly a bit ago and has been undergoing filesystem recovery since. We expect it to be back up within a hour, we apologize for the inconvenience.

Update: Scalise is coming back up now, sites should be back to normal operation shortly. Our administrators are investigating the cause of the crash and will implement fixes to prevent this from happening in the future.


2:42 am

Nipper (MySQL) going down shortly for maintenance

Posted (November 6th, 2008 at 2:42 am PST) by Sandon

The MySQL server ‘nipper’ is going down for some short maintenance at 3 AM PST. The outage should be less than 15 minutes and likely less than 10. I apologize for the late notice about this outage.

-Happy DreamHost Late-Night Team

–Update 11/6/08 3:38am
Drive was replaced, took about 15 minutes. Server is now up. Thanks for the patience.


7:35 am

Credit Card payment errors (RESOLVED)

Posted (November 4th, 2008 at 7:35 am PST) by David

Apparently we are having a problem with our connection to our credit card processing company at the moment. Our admins are working to correct the problem as quickly as possible.

If you have our “AutoPay” feature enabled you can leave things as they are because the system will re-attempt to process your charge weekly until it goes through successfully. We do have a 60 day billing grace period, so as long as your account is settled within 60 days it won’t be suspended.

If you’re trying to make a manual payment please wait a few hours and try again.

We apologize for the inconvenience.

UPDATE: 7:50 PST – It looks like we’ve got this problem resolved. Manual credit card payments are being processed successfully again. We suspect that “AutoPay” payments will go through normally as well. We’re still going to keep an eye on this for a while – just in case.

For those of you whose accounts were up-to-date and had just this previous “AutoPay” payment fail you can leave things as they are and the system will automatically attempt to process the charges next week. However, if you were behind on your payments you should make a manual payment immediately or risk having your account suspended for non-payment by the system.


1:26 pm

Host VSQL Machine Down for Maintenance

Posted (November 3rd, 2008 at 1:26 pm PST) by Kyle

We are bringing down one of our Private SQL host machines to preform critical maintenance. The problem is isolated to the following Private SQL servers:

psmysql1829 psmysql3011 psmysql3166 psmysql4330 psmysql4975 psmysql797 psmysql961 psmysql964 psmysql967 psmysql970 psmysql975 psmysql986 psmysql1861 psmysql3016 psmysql3171 psmysql4576 psmysql744 psmysql799 psmysql962 psmysql965 psmysql968 psmysql971 psmysql978 psmysql991 psmysql1863 psmysql3029 psmysql3228 psmysql4588 psmysql745 psmysql802 psmysql963 psmysql966 psmysql969 psmysql972 psmysql979

Estimated downtime is 20-30 minutes. We apologize in advance for the inconvenience.

Update: Services have returned to normal, sorry about the downtime!


12:02 pm

Last Chance for Charity!

Posted (November 3rd, 2008 at 12:02 pm PST) by Josh

Tomorrow we’re going to change our charities at Home > Charity .. so you better go donate now to The American Red Cross and/or Edu Relief if you want us to match it!

So far we’ve gotten 16 donations to The American Red Cross for a total of $428.00 and 10 donations to Edu Relief for a total of $222.50.

We can do WAY BETTER!


5:19 pm

Martinellis not so bubbly

Posted (November 2nd, 2008 at 5:19 pm PST) by Jason C

Users on the server martinellis may have been experiencing some performance issues over the past few days. While loads would spike occasionally, the server’s performance would level back out and no indicators would point to a user or file server causing problems.

We’ve restarted the server now — and while there are no signs that point to a user or file server causing these issues, we will continue to keep a close eye on the machine. If we need to migrate this server to new hardware, we will either append this post or make a new one.


12:00 pm

Missing address books in SquirrelMail

Posted (November 2nd, 2008 at 12:00 pm PST) by brians

Recent changes that we made to Squirrel Mail caused address books to get removed. We are in the processes of restoring your address books with our backup performed on 2008-10-28. Unfortunately, any entries added after that date will not be included. The restore should be completed within a few hours. Again, we sincerely apologize for the inconvenience this has caused and appreciate your continued patience. Please check back for future updates.

Update 2008-11-02: We had to attempt the restore twice as the sheer number of addresses (over 3 million) broke the script. It’s all finished and tested, the addresses are now back. Sorry about the delay and the inconvenience this has caused.


4:48 am

Hoho is not so cheery.

Posted (November 1st, 2008 at 4:48 am PST) by Sandon

It looks like its OS partition is corrupt possibly due to a hardware failure causing fsck to crash (segment fault). No need to worry though as no customer data is stored on this machine! I am moving it over to new hardware now and services should be restored within an hour.

Happy DreamHost Happy Late-Night Halloween Team!

Update 05:20 AM PDT:
The Hoho is all moved to new hardware and service is restored. Hoho is jolly again! Please contact support if you are still having any issues and are on ‘hoho.’


 
© 1996-2008, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
6:50 pm

Bruno Downtime

Posted (October 31st, 2008 at 6:50 pm PST) by micah

Bruno locked up and had to be powercycled. It’s currently checking it’s file system and should be up shortly. Sorry for the inconvenience!

Happy DreamHost Happy Halloween Team!


9:10 am

barker, spilotro and provenzano outage!

Posted (October 31st, 2008 at 9:10 am PST) by Brian@dreamhost

It looks like these three servers had some issues coming back up after a reboot this morning! Normally, upon rebooting, these machines should come back up within a few minutes, however they seem to need to check their disks before loading up the rest of the operating system.

Provenzano should be back up completely, and spilotro and barker should be back up within the next 15 minutes or so!

We apologize for the problems this caused!

Update 11:26am PST: Spilotro came back and shortly thereafter so did Barker. We have made adjustments so that future checks will be preformed more expeditiously to avoid lengthy downtimes. We apologize for the outage.


10:02 am

some MySQL PS problems

Posted (October 30th, 2008 at 10:02 am PST) by danwy

One of our MySQL PS host machines is acting up right now. One of our admins is currently investigating the problem, but doesn’t have any updates at this point. If you’re on one of the following MySQL PS servers and are experiencing problems, we are investigating the issues and hope to have it resolved as soon as possible:

psmysql3163 psmysql3843 psmysql3851 psmysql3870 psmysql3882 psmysql3888 psmysql3893 psmysql3897 psmysql3914 psmysql3939 psmysql3976 psmysql4021 psmysql4044 psmysql4062 psmysql4068 psmysql4077 psmysql4137 psmysql4139 psmysql4145 psmysql4150 psmysql4155 psmysql4175 psmysql4183 psmysql4238 psmysql4250 psmysql4255 psmysql4260 psmysql4264 psmysql4276 psmysql4281 psmysql4292 psmysql4298 psmysql4310 psmysql4311 psmysql4314 psmysql4315 psmysql4410 psmysql4415 psmysql4416 psmysql4592 psmysql4613 psmysql4634 psmysql4645 psmysql4659 psmysql4712 psmysql4714 psmysql4715 psmysql4765 psmysql4772 psmysql4878 psmysql4890 psmysql4932 psmysql4934 psmysql4936 psmysql4939 psmysql4945 psmysql4992 psmysql5000 psmysql5019

Please check back on this post, as we will be making updates to it as we troubleshoot the problems.

UPDATE 10:30am PST: We’ve tracked this problem down to a bad RAID card on the MySQL PS host. We’re moving the disks over to new hardware now and should have it fixed up soon. We’ll update this post again once everything is resolved.
UPDATE 12:20pm PST: We have swapped the drives to a new chassis and after a bit of reconfiguring the guests are now back up and running. If you continue to have issues with your mysql private server please contact support.


10:52 am

Fileserver problem on Frisky

Posted (October 29th, 2008 at 10:52 am PST) by Kyle

The dreaded fileserver which has been plaguing the Frisky cluster lost a drive this morning and panicked while reconstructing. Our admins are diligently working to resolve the problem with the fileserver and the affected machines. We have already began offloading customers from this fileserver to new machines but there is a large volume of data and it will take time (the offloading started several days ago). We apologize for the repeated issues customers on this cluster are having (cluster = mail server from the account status link on the panel) due to this fileserver, the good news is once we complete the data migration these sort of problems will be a thing of the past.

UPDATE: The file server is back up. The moves we are running to empty this hardware have started up again. Ultimately we will not have full resolution on this recurring problem until the hardware is out of the loop. Initially we had tried replacing parts of the hardware to resolve these problems. Since that didn’t work, we’re emptying it and removing it from active service. Sorry about the problems this has caused you.


9:17 pm

Babysitting Bugsy

Posted (October 28th, 2008 at 9:17 pm PST) by Jason C

It would appear that the server “bugsy” had a few issues that required the loving attention of a member of our admin team.  As it stands, the server is going thru a deep file system check that requires some close monitoring due to possible intervention.  Once it is done with that, everything should hopefully come back up and return to normal.  If that doesn’t happen, we’ll be sure to post an update here.

UPDATE: We have been working on Bugsy throughout the evening and we are at a point where we are working on repairing the disks.  If that does not work, we’ll restore from backups and then repair the disks.  Unfortunately, we have no ETA on this process, but we will update here as soon as we know more.

UPDATE 10/29 3:10p Pacific: Unfortunately we still do not have a solid ETA on a resolution for bugsy. We are however attacking the issue with both solutions listed above. We have a new bugsy up and we are copying data from our backups to this server. The new bugsy is the live server so you may notice that your files are missing. Do not fear, they are being copied from backups. We are also still trying to repair the disks on old bugsy. If that is successful then old bugsy will go back live. So we’re basically in a situation where the first solution to finish will be the one we go with.

Update 10/30 11:00a Pacific: The data being copied from backups to the new Bugsy is roughly 50% done. This unfortunately means that we are likely looking at more than 24 hours for the moves to be complete. As your files are moved your sites will be working, but we unfortunately do not have a way to prioritize the moves by account at this time. The disk repair on old Bugsy was not 100% successful unfortunately. The plan is to let the copy of the backups finish, then run another copy of data on old Bugsy to the new one. The backups we have are unfortunately 2 weeks old, so after we get that data up we will run the second copy in order to get your most up to date files. Sorry about the problems this is causing you. A free month of hosting will be added for customers on the Bugsy server.

Update 10/31 3:30p Pacific: The data moves from the 2 week old backups to the new server have finished. We are now starting the data move from old Bugsy to the new Bugsy. Those of you mentioning you out like to opt-out on the second restore have been set to skip. We are also not going to overwrite any files currently on the server if they have a newer date than the ones copying over. So your sites should be up now. Sorry about the problems this has caused you.


6:23 pm

Not Feeling So Frisky

Posted (October 27th, 2008 at 6:23 pm PST) by Jason C

One of the central network file servers on the frisky server cluster was giving us a little trouble and may have caused the following servers to go sideways:

barracuda beaker boxster bridgatchina bugatti bumper cadillac camaro catalyst cayman chennai chevelle circatreews01 colintsui coneybeare continental corvette creativeliberations cutlass dart democracy dmc edsel eirenemachine elcamino eldorado elise enzo eredux esprit fairlane ferrari fiat firebird fozzie funty1 galaxie gdr gibble godfuel guru highground hpl hudson hummer hupmobile impala jaguar jetta jonlin knocker kyfserver labnol lesabre logichost malibu marcianos mclaren merkur miata model-t montecarlo murcielago ninja nova olds ontrack packard palmea passat phaeton phiya pierce pinto probe prowler ps3649 ps3718 ps3796 ps3995 ps4207 ps4224 ps4364 ps4389 ps4418 ps4566 ps4602 ps4707 rambler rawstock renault rolls sciencewerk scion scthreads sexylinker shelby sinagoo skylark smokelabs sos spyder sten stingray stratus studebaker tempest tesla thegibson thog thunderbird titan tomkincaid touareg towncar trans-am veyron vsuno wcweb1 wisdev workserver wrangler xaevir xcopi yonder zawi

If you’re on one of the servers listed above and have been seeing recent issues with your sites, things should be leveling out now.  There are a handful still going thru the reboot process, so please be aware that things aren’t fully back to normal just yet — but they should be soon.


4:36 pm

Panel and Webmail maintenance

Posted (October 27th, 2008 at 4:36 pm PST) by DreamHost

We will be taking the DreamHost Web Panel, and DreamHost-provided Webmail off line for maintenance this Thursday, at 10PM PDT. The maintenance window for this is 2 hours extending until 11:59PM, expected downtime within that window is 1 hour.


3:15 am

DreamHost PS problem on spacey.

Posted (October 27th, 2008 at 3:15 am PST) by DreamHost

DreamHost PS customers in the spacey cluster may see a bit of downtime. A hard drive failed on one of the DreamHost PS hosts (#22) and that host is moving to new hardware. Sorry for the interruption.


3:00 pm

Filer on the fritz (back online!)

Posted (October 24th, 2008 at 3:00 pm PST) by Leon

One of our filers, scraper, had his poor widdle feewings hurt and has decided to stop working. Affected machines are:

achilles aviation backside badillo baht bazooka blueprint boppity boyd caramello centipede cheeto cheezit crack cruisin dinero dorito formosa go gummi happy herod hillhurst hillmont hoho hyperion ikari ike jaedyne jalapa julie jurupa katamari kitkat knocker larchmont lei limbo-randy look moorpark munchies nash ninja orbital outside payday pernod pineapple powerbar pretzel pringle randy-vserver07 randy-vserver09 randy-vserver11 rhod riyadhorg rollo roscoe rossmore selma shocktart skittle slauson spree spunky-vserver19 spunky-vserver20 tootsie twinkie ugh venice vermouth vex virgil westside wilmington wilshire yoda york

Our esteemed admin team is on it. Sorry about that!

Update 4:19 Pacific: The filer has been fixed, it appears to have been a cabling or HBA issue on one of the disk trays. We are now running around rebooting the various servers it crashed, service should be restored in about 15 minutes. Please let support know if your stuff is still down after that!


10:58 am

Network hiccup

Posted (October 24th, 2008 at 10:58 am PST) by andrea

One of our network providers went down for a few minutes but one of the other ones jumped in shortly after to pick up the affected connections. Things were a bit slow and gappy because of that, but the failed network is already working again. Things should be back to normal, but do let us know if you’re still having problems connecting. Please provide details, so we can better assist you.

Sorry about all the inconvenience this has caused.


 
© 1996-2008, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
11:20 am

Router reboot this Friday night

Posted (October 23rd, 2008 at 11:20 am PST) by DreamHost

This Friday, October 24, 2008 at 10:15pm Pacific time we will be performing an OS upgrade on our core router. This requires a reload, during which time all DreamHost services will be unreachable for about 15 minutes. After this time everything should be back to normal.

This is related to:

http://www.dreamhoststatus.com/2008/10/22/network-issues/

http://www.dreamhoststatus.com/2008/09/21/major-network-issues/

Cisco’s technical support is not certain this will fix the issue, however they advised that our current version of IOS is deferred due to many related bugs. We are hopeful this will make this problem go away, or at least less prevalent.

Edit: 10:14 PM Pacific Rebooting in about 1 minute! See you in 15!

Edit: 10:34 PM Pacific We’re back! The router is still thinking quite hard about things, but based upon blinking hard disk lights, and my general ability to update this post, we’re more or less back online! I will be going through and making sure all of our servers are up and running. Please contact support if you have any lingering issues.


10:18 am

network issues

Posted (October 22nd, 2008 at 10:18 am PST) by kitchen

we are currently experiencing some connectivity issues at our primary facility, which is causing access issues for all of our customers. We have our network folks working on it and expect a resolution soon.

We will post updates as things progress, and thank you for your patience while we resolve this matter.

Update 1:50pm: We ran into a Cisco bug which was causing the problems. We opened a bug report with them, and waiting to hear back on how to proceed. Please check back for further updates.


11:10 am

Jebediah network card wonked

Posted (October 20th, 2008 at 11:10 am PST) by Kyle

The network card in jebediah isn’t cooperating this morning despite our valiant efforts. The server is being moved to new hardware, approx. downtime is 30 minutes.

Update: Failover went smooooothly and everything is slowly returning to normal, if problems are persisting please contact support.


9:24 am

Issues with accessing databases in phpmyadmin

Posted (October 16th, 2008 at 9:24 am PST) by andrea

Some customers on certain database machines have been having issues accessing their databases using phpmyadmin. We apologize for the inconvenience this is causing. Our admin team is looking into the cause and is working on a resolution. The symptoms of the problem are one of the following: blank page loading instead of the phpmyadmin interface, some interfaces are missing images and/or the dbs are not selectable in the drop down list. For the latter one we are suggesting a workaround for now: work in phpmyadmin without using the left frame, browse to ‘Databases’ on the right, selecting th one you want.

Command line database access remains unaffected. As a reminder, here is the generic command:
mysql -u username -ppassword -h hostname databasename

Please check back here for updates, and sorry again for the problems.


10:55 pm

Problems on Kray

Posted (October 15th, 2008 at 10:55 pm PST) by Ralph

The problem is requiring us to move all users and their data off of the machine. Once we have you moved off of Kray service should be restored for you. If you do not know what hosting machine you are on, please check the ‘Account Status’ link in the DreamHost webpanel. It is in the upper right corner, just click on it and the ‘Your Web Server’ information is your hosting machine. Sorry about the downtime.

Update: Users with their home machine previously on Kray should be fixed now. We actually reverted the move since Kray was full of users who were recently moved to the machine. We finished this last night about 4 hours after this post was made.


10:42 pm

Frisky Mail Problems

Posted (October 15th, 2008 at 10:42 pm PST) by Ralph

We are currently experience a mail outage with Frisky mail. If you are unsure if you are in the frisky cluster please log into the web panel and click the ‘Account Status’ link on the top right of the panel. The Email Server will say Frisky. Mail will be queued up for delivery instead of bounced. Once we have the authentication database populated again mail service will return. We have an ETA of about 2 hours. Sorry about the problems this has caused.

Update: Mail service has returned to normal. The ETA listed above ended up being roughly when we got service restored.


10:01 am

File Server Issue Causing Downtime

Posted (October 15th, 2008 at 10:01 am PST) by mir

Our Admin team is working on a problem with one of our file servers. The servers mounted to that file server may be experiencing degraded performance until we have this fixed:

athena barracuda belair boxster bugatti cadillac camaro carlititos carrera cayman chevelle continental corvette cutlass dart delorean edsel elcamino eldorado elise enzo esprit fairlane ferrari fiat fiero firebird galaxie gremlin highground hornet hudson hummer hupmobile impala jaguar jetta lancer lesabre malibu manmeat mclaren merkur miata model-t montecarlo murcielago nova olds pacer packard passat phaeton pierce pinto probe prowler rambler rawstock renault riviera rolls scion shelby skoda skylark stingray stratus studebaker tempest tesla thunderbird titan touareg towncar trans-am veyron wrangler

Update: we understand that this has been happening too frequently so the next step is to replace the head unit entirely with new hardware. We’ll begin this process shortly and update again when that is completed and the machine is back online (we anticipate this to be a permanent fix).

Update: the admin team has reported that the hardware swap was successful and the sites that we have been testing are back on line but we’re monitoring the situation closely and looking for any servers that may need additional attention.


1:00 am

Lipton moving to new hardware

Posted (October 14th, 2008 at 1:00 am PST) by DreamHost

Lipton seems to be having a problem with its disk (customer data should be fine), and it needs to be moved to new hardware.

Update: It was able to be saved without being moved over. Services should be running normally.


1:26 pm

Brisk getting new hardware

Posted (October 13th, 2008 at 1:26 pm PST) by kitchen

Yes, we are well aware we just gave brisk some new hardware yesterday, however its new hardware is having issues as well, so we’re going to give it another machine, again.

This process should take about 30-45 minutes and we will post an update here once it’s all done!


6:22 am

Baachus consumed too much wine.

Posted (October 13th, 2008 at 6:22 am PST) by Kyle

The god of wine himself was having a Monday morning hangover and despite several attempts with Gatorade, and a ‘lil hair of the dog it looks like it’s time to move the machine to new hardware. Estimated downtime is one hour, we’re sorry for any outages resulting from this incident.

Update: The new Baachus being a bit cranky and is taking a bit longer than expected, our apologies

Update: Configuration updates are finishing on Baachus and some apache services are already back up and kicking, everything should be fine in the next few minutes if not already.

Update: Looks like we were not quite out of the woods, Baachus is being finiky, we’re investigating the cause to get this resolved asaprightaway.

Update: Baachus is back doing it’s thang, if you are still having troubles please contact support.


 
© 1996-2008, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
5:09 pm

Brisk ‘s memory is a little slow

Posted (October 12th, 2008 at 5:09 pm PST) by glen

Web server Brisk looks to be having hardware troubles with it’s RAM. We will be moving it over to new hardware immediately. Downtime likely 30 minutes to 1 hour. Updates to follow.

Updated
Web services half up. The rest should should be following in the coming minutes. Please contact support if you have any further issues. This would not have affected any other servers besides the Brisk web server.


12:04 pm

Mail Service Down

Posted (October 10th, 2008 at 12:04 pm PST) by brians

We are experiencing problems with our mail service at the moment. This appears to be affecting all clusters. Our admins are currently investigating this matter and hope to have the issue resolved shortly. We sincerely apologize for the inconvenience this may have caused and appreciate your continued patience. Please check back for future updates.


Update Sat October 11th, 2008. 12:02 PM PST
This issue was resolved minutes after the initial post. Our admins had to restart courierauthd on all mail machines on our network. Sorry for the delayed update.


2:33 pm

We’re So Pretty

Posted (October 9th, 2008 at 2:33 pm PST) by Jason C

Some of you might have noticed the new coat of paint here on the status blog.  For those of you fretting about any issues you might currently be having — or how a new theme doesn’t really help your sites get back up any faster — please don’t.  The Support Team is seriously working hard at sorting thru the open tickets in the system and resolving them as quickly as possible while I write this entry.

We just figured that most of you would appreciate something a little cleaner (see: modern) than what was here in the past.  Hopefully you find it to be a bit more readable and generally nicer to deal with.  Should you notice any problems with the new theme tho, let us know in the comments!


7:57 am

Spacey-vserver17 kicked the bucket.

Posted (October 4th, 2008 at 7:57 am PST) by Sandon

It looks like its PSU has died and is not turning on at all. If you have a PS which is named any of the following:

cupula, fpccprod, glyn0x001, keiwc, ksharpie, ps3673, ps3772, ps4162, ps4185, rocky, samhall, servebooz, siriusb, superman, uiopluss, xecova

Then this outage is effecting you. This should not be effecting anyone on a PS other than the ones listed above. I am starting the fail-over process to move everything to a new server now and it service should be restored in approximately one hour or less.

Update 8:52 AM PST: All services to the above PSs should be restored. If you are still having any issues connecting to your sites or other services please contact support!


6:35 am

MySQL Service Downtime

Posted (October 2nd, 2008 at 6:35 am PST) by Justin@DH

We are experiencing some mysql service downtime at the moment. We are still investigating the cause and working to get this fixed up as soon as possible. It would seem a system-wide configure of the mysql services is what triggered this, with new configuration files not being pushed to the new servers. We do have our admins on the job and will do everything we can to get service restored. Sorry for the problems. When we have further info we will update this status post.

Update: I apologize for the lack of a followup but we actually resolved this Thursday morning a short time after the initial posting of this status report (the admin who did was unable to post at that time).


1:39 pm

Eradicator troubles

Posted (October 1st, 2008 at 1:39 pm PST) by Leon

One of our filers, eradicator, is having some issues right now. We have admins checking it out currently. We’ll advise when this is working normally again. The affected machines are:

barracuda beaker bolsa boxster bridgatchina bugatti bumper cadillac camaro catalyst cayman chennai chevelle circatreews01 colintsui coneybeare continental corvette creativeliberations cutlass dart democracy dmc edsel eirenemachine elcamino eldorado elise enzo eredux esprit fairlane ferrari fiat firebird fozzie funty1 galaxie gdr gibble godfuel guru highground hpl hudson hummer hupmobile impala jaguar jetta jonlin knocker kyfserver labnol lesabre logichost malibu marcianos mclaren merkur miata model-t montecarlo murcielago ninja nova olds ontrack packard palmea passat phaeton phiya pierce pinto probe prowler ps3649 ps3718 ps3796 ps3995 rambler rawstock renault rolls sciencewerk scion scthreads sexylinker shelby sinagoo skylark smokelabs sos spyder sten stingray stratus studebaker tempest tesla thegibson thog thunderbird titan tomkincaid touareg towncar trans-am veyron vsuno wcweb1 wisdev workserver wrangler xaevir xcopi yonder zawi

We will update as soon as we know what’s going on.

Update – 3:40 PM PST: The network file server was crashing with errors from its PCI bus.  Because of this, data corruption protection triggers were tripping.  Our admin team worked together to quickly swap out the motherboard and RAM assembly and reseat the cards and the problem appears to have gone away.  Services are coming back up now and problems should stabilize completely in the next half hour or so.


6:05 pm

Hades burning!

Posted (September 29th, 2008 at 6:05 pm PST) by Leon

Anyway, we’re having some major issues with hades (I know, I know, it’s not just been today) so we’re moving it to new hardware. It should be back up in about an hour or so, hopefully not much longer. We’ll keep you updated.

Update Mon Sep 29 19:20:53 PDT 2008: Hades is back up and running normally.

Update Wed October 1st, 2008 at 12:38 am PST: The new hades appears to be having some trouble with its RAM, so it is being moved over to new hardware.

Update: it went smoothly and hades is up now.

Update Fri October 3rd, 2008. 3:31am: Looks like either hades has the worst luck in the (under)world, or there’s a different problem masquerading as hardware failure. Currently being investigated.


10:06 am

Homie-vsql02 is down

Posted (September 27th, 2008 at 10:06 am PST) by DreamHost

Homie-vsql02 is currently down, the datacenter has been called and a representative is on their way to reboot it.  I’ll update this post as soon as things are back up.  I’m very sorry for the inconvenience!   The following PS MySQL guests are affected:

psmysql3353 psmysql3347 psmysql3341 psmysql3345 psmysql3339 psmysql3340 psmysql3366 psmysql3334 psmysql3835 psmysql3680 psmysql3327 psmysql3354 psmysql3738 psmysql3350 psmysql3336 psmysql3338 psmysql3834 psmysql3346 psmysql3352 psmysql3333 psmysql3343 psmysql3349 psmysql3328 psmysql3332 psmysql3330 psmysql3357 psmysql3803 psmysql3729 psmysql3690 psmysql3697 psmysql3693 psmysql3771 psmysql3813 psmysql3351 psmysql3809 psmysql3802 psmysql3728 psmysql3625 psmysql3694 psmysql3691 psmysql3732 psmysql3695 psmysql3337 psmysql3692 psmysql3734 psmysql3830 psmysql3727 psmysql3804 psmysql3816 psmysql3684 psmysql3363 psmysql3814 psmysql3832 psmysql3815 psmysql3687 psmysql3810 psmysql3344 psmysql3831 psmysql3805 psmysql3682 psmysql3681 psmysql3696

Update: Homie-vsql02 is back up.  If you continue to have any troubles with this, please contact Technical Support.


7:31 pm

Mills moving to new hardware…sorta.

Posted (September 26th, 2008 at 7:31 pm PST) by Leon

A bit of miscommunication has caused some problems with mills. It’s being set up on some new hardware as we speak and should be back within the hour. Sorry about that!

Update, 9:03 PST: Sorry that took so long, ugh! Mills is back up and running. Please contact your super-efficient and friendly Dreamhost Technical Support Person for any remaining issues!


1:07 pm

Webserver “Tootsie” offline

Posted (September 25th, 2008 at 1:07 pm PST) by jeremy

It didn’t take so well to the brief power outage in its rack and we’re working on getting it on new hardware right now. We expect to have the server back up by 1:30pm (PDT).

Update: 2:25pm

We apologize for the extra time it took to get the server back up. It turned out that the outage also damaged the spare server we had ready so we had to rebuild the hard drive in order to get the server running again. At this time, about half of the websites hosted on tootsie are back up and the rest should be up in mere minutes.


 
© 1996-2008, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
10:57 am

Power strip / switch outage in one of the spunky racks

Posted (September 25th, 2008 at 10:57 am PST) by kitchen

One of the power strips or switches servicing the spunky cluster appears to be down. We are currently investigating the issue, however the following machines are currently inaccessible:

fletcher grappa fairfax bubbleyum tootsie beaudry limbo-spunky3 nash butler payday dorito planters caramello jefferson starburst licorice plenty lifesaver aero hyperion truffle zagnut

As we get more information about the issue we will post updates here.

Thank you for your patience.

Update 11:17am Pacific: It turns out one of the breakers popped in the datacenter’s electrical panel. They have since restored the breaker and we will be on the look out for further issues. We’re fairly confident we did not cause any spikes on our side of the system, all of our hardware is running normally, and we’re only showing 12amps RMS on our power strip. It is a 20 amp circuit, and is thus well below the 16 amp alarm threshold on our power strip.
RPC-10>status

-------------------------------------------------------------------------------
| Outlet | True RMS | Peak RMS | True RMS | Average | Volt- |
| Group | Current | Current | Voltage | Power | Amps |
-------------------------------------------------------------------------------
| Outlet 1-20 | 12.1 Amps | 12.9 Amps | 116.7 Volts | 1342 Watts | 1380 VA |
-------------------------------------------------------------------------------


11:14 am

More filer problems on the frisky cluster

Posted (September 23rd, 2008 at 11:14 am PST) by danwy

In addition to the filer problems we’re having on janky, one of the filers on frisky also started acting up a short time ago. This is causing issues on several of the servers on the frisky cluster. Our admins are looking into this filer as well as the one on janky and hope to have both of them back to normal as soon as we possibly can. We’re sorry for the inconvenience this causes you.

UPDATE: 12:32pm (PST) – We have been unable to find the cause of these filer issues on Frisky. Our admin is temporarily taking down the filer to run further diagnostics on it to find the problem. Unfortunately, we don’t have an ETA at this point, but we are doing everything we possibly can to get things back up and running ASAP.

UPDATE: 1:57pm (PST) – We’ve seen a few questions come in wondering which servers are affected by this filer.  If you’re on one of the following servers, you will most likely be seeing problems: 

barracuda beaker bolsa boxster bridgatchina bugatti bumper cadillac camaro catalyst cayman chennai chevelle circatreews01 colintsui coneybeare continental corvette creativeliberations cutlass dart democracy dmc edsel eirenemachine elcamino eldorado elise enzo eredux esprit fairlane ferrari fiat firebird fozzie frisky-vserver01 frisky-vserver02 frisky-vserver03 frisky-vserver05 frisky-vserver06 frisky-vserver07 frisky-vserver08 frisky-vserver11 frisky-vserver12 frisky-vserver13 frisky-vserver14 frisky-vserver15 frisky-vserver16 frisky-vserver17 frisky-vserver18 frisky-vserver19 funty1 galaxie gdr gibble godfuel guru highground hpl hudson hummer hupmobile impala jaguar jetta jonlin knocker kyfserver labnol lesabre logichost malibu marcianos mclaren merkur miata model-t montecarlo murcielago ninja nova olds ontrack packard palmea passat phaeton phiya pierce pinto probe prowler ps3649 ps3718 ps3796 ps3995 rambler rawstock renault rolls rtryyuy sciencewerk scion scthreads sexylinker shelby sinagoo skylark smokelabs sos spyder sten stingray stratus studebaker tempest tesla thegibson thog thunderbird titan tomkincaid touareg towncar trans-am veyron vsuno wcweb1 wisdev workserver wrangler xaevir xcopi yonder zawi

More updates (and hopefully a resolution) will be on the way very soon.

UPDATE: 4:27pm (PST) – The latest word from our admin team is that this issue was caused by a bad backplane on one of the filer shelves.  That made things a bit hard to narrow down, but we’ve found it now and have remedied things.  The file server is back up now tho and it’s rebuilding a disk.  Still, most customer sites on frisky that were seeing problems because of this should be taken care of now.


10:06 am

Filer causing issues for Janky customers.

Posted (September 23rd, 2008 at 10:06 am PST) by mir

We’re in the process of correcting this – one of the central file servers on the cluster ‘janky’ is having issues which is resulting in high server loads and disruption of service on all of the hosting servers which mount data from that file server (which is almost all of the machines on that cluster). The admin team is fixing this and we’ll post again once it is resolved (a quick reboot of the effected shared hosting machines once we fix the filer will get things back to normal).

UPDATE: 12:35pm (PST) – One of the drives on this filer is in the process of being rebuilt. It should be back up and running shortly. We’ll update this post once it’s done.

UPDATE: 2:27pm (PST) – The rebuild of the drive is still chugging along.  Just so customers aren’t left in the dark, here’s a list of servers that are running into problems due to this issue:

adonis agamemnon ajax aleph aphrodite apollo aries baachus battle blackcat brandon calypso chronos contactrene coolblue criollo daedalus dexau dione dionysus dlc electra erida eros etoffemakaraps evilfree fleur gencer gladys glaskonia gorgon hades hecaee hecate hector helios hephaestus hera heracles hermes hestia hypnos icarus ifxchina janky-vserver01 janky-vserver02 janky-vserver03 janky-vserver06 janky-vserver07 janky-vserver08 janky-vserver09 janky-vserver10 janky-vserver11 janky-vserver13 janky-vserver14 janky-vserver15 janky-vserver16 janky-vserver17 janky-vserver19 janky-vserver20 jdp kaskabot katana kermit knocker lakeview lasagonia leto locomotiva manzanas maxxdidit medusa midas minerva morpheus morrow nemesis ninja noteflight octane oedipus omatech omid osiris pan pandora pedestriantv planfour poseidon proteus provancetek ps3654 ps3988 rhea roo saasy sairam showrock simoserv sixtyfour sports statler supermario tailfin tartarus tethys thanatos trackereactor tripleoptic triton tyche webhostservice wotan xinetau yu zeetab zeus

More updates to this entry as they bubble to the surface.

UPDATE: 10:02pm (PST) – The drive rebuild has finished and servers utilizing this file server should stabilize.  If you continue to see issues, please let our support know.  We want to make sure everyone seeing problems gets taken care of — so anything we can do to help, we gladly will.


6:23 pm

Major network issues

Posted (September 21st, 2008 at 6:23 pm PST) by DreamHost

We are currently experiencing problems with one of our network providers. Our network admins are already on the case and working to get this fixed as soon as possible. More updates as information becomes available.

Update: 18:28: This seems to have already recovered. We’re watching all of our servers to find any that may have not liked having their internet taken away. If you’re seeing any problems with sites hosted with us please refrain from contacting support for the time being.


8:10 pm

Some Homie Private Servers Are Down

Posted (September 18th, 2008 at 8:10 pm PST) by Jason C

It would appear that we had a bit of an issue with one of the Private Server hosts that required our on-call admin to head over to the datacenter to have to give it some TLC.

The Private Servers encountering issues because of this are as follows: ps3717 ps3723 ps3741 ps3742 ps3744 ps3745 ps3747 ps3748 ps3749 ps3750 ps3751 ps3752 ps3753 ps3754 ps3755 ps3756 ps3757 ps3758 ps3760 ps3761 ps3763 ps3764 ps3768 ps3769 ps3770 ps3774 ps3775 ps3780 ps3782 ps3783 ps3787 ps3791 ps3795 ps3800 ps3821 ps3824 ps3827 ps3846 ps3848 ps3857 ps3861

As more information is discovered about this issue, we will update this entry.  Please accept our most humble apologies for any service outages this might be causing you.


10:03 am

Emergency Database Maintenence!

Posted (September 18th, 2008 at 10:03 am PST) by Justin@DH

Our central database server requires some immediate attention and it is currently down for maintenance. The panel will be unavailable until this is resolved, but we are working to get this done as soon as possible. Sorry for the late notice, this should fix up the slowness with the control panel as well.

— Happy DreamHost Database Maintenance Team

— Edit: Thu Sep 18 10:39:24 PDT 2008

The server is back online and services restored!


7:42 am

Control Panel Slowness

Posted (September 18th, 2008 at 7:42 am PST) by David

Some of you may have noticed that the control panel (https://panel.dreamhost.com/) is running slower than normal. This is true. We’re looking into this and hope to have it back to normal speed as soon as possible. We apologize for any inconvenience.

The not-so-happy, is it Monday or is it Thursday? support team.


2:58 am

Some dreamhostps may be experiencing issues.

Posted (September 18th, 2008 at 2:58 am PST) by DreamHost

One of our ps hosts is having issues that require it to be moved to new hardware. The following ps are affected:
brakar
gdr
persia
pythius
seawxhost
sourcetone
sten
stupidbird

Update 4:35 PDT

Sorry for the delay! There were some complication with the new hardware which caused some dealsy. All of the above servers are up and everything should be working properly now. If you see your PS listed above and are still having issues please contact support!


3:51 pm

Rails 2.1.1 is coming!

Posted (September 16th, 2008 at 3:51 pm PST) by Terri

It seems like just yesterday we updated to 2.0.0, and of course there’s a new Rails already! I’m starting to build packages for 2.1.1 today, and I’ll then slowly test and roll the new version out over the next week. So, once again, freeze your rails if you haven’t already.

Freezing is easy! You need to ssh into your account, cd over to project, then run: rake rails:freeze:gems – and that’s it!


2:21 pm

Janky Locales

Posted (September 12th, 2008 at 2:21 pm PST) by Kyle

Locales were updated across the janky cluster and it has caused some unexpected issues. The problem should be remedied shortly. You may notice elevated loads if your home machine resides in this cluster. To determine which cluster you are in you can click the Account Status link on the web panel and look at which “Mail Machine” it displays.


 
© 1996-2008, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
10:12 am

larchmont eats dirt

Posted (September 12th, 2008 at 10:12 am PST) by Kyle

The server larchmont is on the fritz this morning, tanking out on 3 occasions. The server is being failed over to new hardware, estimated downtime is one hour.

Update: Taking a bit longer than normal, fighting to get everything happy again

Update: It’s alllll better now! If you are having any lingering issues please contact support

Update 3:50pm PDT: Looks like the new machine it was placed on took a turn for the worse too! We’re now moving it to another machine. Sorry folks!

Update 5:56pm PDT: It’s finally back up after some issues moving it to the new new server! If you’re still experiencing issues, please contact support.


1:15 pm

Midas’ touch not so golden

Posted (September 10th, 2008 at 1:15 pm PST) by Leon

Sorry, midas users! Looks like the NIC on your machine died. I’ve been moving it to new hardware for about 30 minutes, and it should be back in like another 30 or so. Sorry for the late notice!

Update: the machine was back up as of 3pm, however it was jsut rebooted because of load issues. We’re tracking down who is causing the high load on the machine currently.

Update again: Everything back to normal!


5:13 pm

New WordPress one-click security update

Posted (September 8th, 2008 at 5:13 pm PST) by jeremy

There’s a potentially important security issue in WordPress versions prior to 2.6.2 that may allow malicious attackers to change (and even possibly capture!) the password for any user. Our one-click system has already been updated and we urge everyone to upgrade ASAP.


1:20 am

Skip, skipped into a brick wall

Posted (September 7th, 2008 at 1:20 am PST) by Kyle

The MySQL server Skip is having trouble with its array, it hasn’t been able to recover after reboot. A admin is driving out to the datacenter to address the issue. Estimated time down is unknown at this time however when we have a ETA this post will be updated.

Skip has been resuscitated and is back to doing what it does best.


7:26 pm

kegerator ran out of beer

Posted (September 6th, 2008 at 7:26 pm PST) by kitchen

One of our file servers “kegerator” suddenly stopped serving nfs ops and is currently causing domains hosted on it to be inaccessible. We are looking into this matter and hope to have a resolution soon. This will affect machines and DreamHostPSes on the spacey cluster, including:
7up alpine amp aquafina aremes arizona bambi bawls binamedia bluthton boba bpsites cactuscooler calpico cappuccino caprisun chai clamato coffee coke crush crystallight culligan dads danjacobs derby dietrite digitmidia django downside drpepper dymweb eggnog espresso evian fanboy fanta flipside flowsnisit fpccprod frigate fuze gatas gatorade geyser gingerale grapico hiball horchata hpalumni ibc idmo inside isabellaval izze jolt knocker koolaid krillsc lipton macnimble madrid martinellis milk mimpi minutemaid mmgllc monster mountaindew mrpibb mug musclemilk mycoffee mystic nehi nesbitt ninja optimizer orangebang orvad pellegrino pepsi pixy plainprograms planetapera pom powerade propel raoul rdp refresco riptide rockstar rootbeer seltzer sergnet server1 shasta sigholiday silk smoothie snapple sparkletts stewarts storyhole sunnyd supergate tab tampico tang tea tizer tonic tropicana v8 vanguardps voss welchs widerfunnel xecova yoohoo zoot

We will post updates as things progress!

Update: We fixed the problem about 10 minutes ago (right about the same time this post went up) and have been keeping an eye on it since, and it looks to be ok. There was a runaway cronjob creating tons and tons of files as well as the weekly disk scrub, which, when combined, caused i/o on the file server to peg, slowing things down substantially.


1:18 pm

Pound getting a new network card

Posted (September 5th, 2008 at 1:18 pm PST) by Kyle

Pound has been having issues recently that have required reboots due to a finiky networking card. I’m plopping a new one in there, it shouldn’t take longer than 20 minutes. The end result should be greater stability!

New card installed!


8:14 pm

Janky, Debian Upgrades and You

Posted (September 3rd, 2008 at 8:14 pm PST) by Jason C

 

There have been some upgrades across the janky cluster that have caused a few issues like 500 errors and site downtime.  Because of the volume of support this has created, we thought it would be a wise idea to roll these issues up into a blog post in order to address everything that has been going on.  It’s not a perfect solution by any means — especially since this should have been addressed sooner — but I’m hoping that most of the folks on janky can benefit from the solutions I will outline below.

For the customers who have been seeing temporary site outages between the hours of 10pm and 10am PST, you have been witnessing the upgrade happening itself.  We’re taking all of the servers on your clusters and giving them the latest and greatest version of Debian.  This means that services might error out for a few minutes while new key packages are put into place of old ones.  This is also being done in controlled batches in order to minimize outages and incoming support.

Yes, this strategy isn’t optimal.  As a company that hosts customers around the world, it is hard to pick a time that won’t inconvenience somebody.  As a large percentage of our customer base is in the United States, the times listed above were picked.  This also allowed for a minimal amount of disruption.  Still, people are seeing problems — and we’re sorry if you’re one of them.

If the outages weren’t enough, there have been 500 errors that have been cropping up.  These can be caused by a couple of things:

1. Software (PHP, Python, etc.) that was compiled against the old 32-Bit libraries on our server.  Since the new version of Debian (Etch) is 64-Bit, this has caused locally compiled binaries to fail.  A quick way to fix this is to recompile any custom software against the new libraries on our server.  Unfortunately, since custom software falls under the “unsupported” category, the best we can do is point you at the following DreamHost Status blog post from a few night ago and wish you luck:

http://dreamhoststatus.com/2008/09/02/debian-upgrades-and-custom-php/

Good luck!

2. Recent changes to Apache 2 caused anybody using AuthGroupFile in their .htaccess to get errors.  It turns out this was an oversight.  The module was recently added to Apache 2 and our servers have been reconfigured — so this should be fixed now.  If you continue to see errors tho, you might want to attempt commenting out any AuthGroupFile lines in your site’s .htaccess files (adding a “#” to the beginning of the line should do the trick) and reloading your page.  If your site comes up after commenting that line out, please drop me a line so I can fix your Apache service correctly.

Should you find that your problem was not addressed here and you still need assistance, please let us know right away by contacting Technical Support.  One of our techs will be glad to assist you with any issues that you might be seeing.  We want to make sure that everything is how it should be, after all…

 


5:20 am

Altoid died!

Posted (September 3rd, 2008 at 5:20 am PST) by Sandon

It looks like the web-server ‘altoid’ has bad memory in it which is causing it to crash shortly after booting. I am failing it over to new hardware now. The process should be completed within 1 hour. This is only effecting people whos web-server is listed as ‘altoid’ in the panel.

Update 5:33 PDT:

Actually since its much quicker and I am pretty sure the issue is just ram I went ahead and just replaced the ram in the system and it is now back up and running. Sorry for any inconvenience this caused!


1:09 am

MySQL maintenance 09/04

Posted (September 3rd, 2008 at 1:09 am PST) by Sandon

The following MySQL servers will be going down at 2 AM PDT on September 4th for maitenance. Total downtime per server should not exceed 10 minutes and all servers should be completed within 1 hour so 3 AM PDT. Here is the list of servers:

bubs buck cheshire felix fin flipper morris nobnin riffraff tigger

Update 09/04 2:35 AM PDT:
All services should be restored. Everything went off without a hitch and took less than 40 minutes. Each sever was down for less than 5 minutes. Please contact support if you are still having any connecting to MySQL and your database is on any of the above servers.


7:04 pm

Server schultz ftp down briefly

Posted (September 2nd, 2008 at 7:04 pm PST) by jeremy

To fix up a small problem with the initial creation of the server, we’ll be needing to leave FTP off briefly. Everything should be back up and running normally by 7:30PM PDT.


 
© 1996-2008, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
11:03 am

debian upgrades and custom php

Posted (September 2nd, 2008 at 11:03 am PST) by danwy

We are currently in the process of upgrading our servers to the newest version of Debian, etch. If you have custom php compiled against the old 32bit libraries your site will no longer function correctly. If you are running custom php and start getting errors like these in your domains error.log, this is the cause:

php.cgi: error while loading shared libraries: whatever.so.15: cannot open shared object file: No such file or directory

You are going to need to recompile your php against the new 64bit toolchain, since it was more than likely compiled against the old 32bit libraries. This is not something that we can offer support with, but you can find more information about how to compile PHP in our wiki:

http://wiki.dreamhost.com/Installing_PHP5#Compiling_a_Customized_PHP_5


11:24 am

Vidar Problems

Posted (August 30th, 2008 at 11:24 am PST) by DreamHost

Vidar as been experiencing some off and on reboots due to memory problems. We have recently upgraded the amount of memory in the machine and we have also addressed some user script problems that were consuming too many resources. With this we believe the machine will become more stable now, however we’re still keeping an eye on it to ensure there are no lingering problems.


9:29 pm

File Server Upgrades 2008-08-28

Posted (August 28th, 2008 at 9:29 pm PST) by Patrick

Home stretch for this week! Starting at 22:30 is fozzie, followed by thog, then gladys if time permits.

Update @01:05

Fozzie is three quarters of the way through the upgrade process. Thog is at 10%. Both should be back up by 02:30. After which, we’ll start on gladys. For those on the frisky cluster, once thog is done, your upgrades will be complete.

Update @01:33

Fozzie is back up. Thog is chugging along and is 70% through the update process.

Update @02:08

Thog is back up and running! Last on the list for tonight is gladys, which will begin around 02:30.

Update @04:55

Gladys is serving files again! Good night!


1:13 pm

Frisky Network File Server Issues

Posted (August 28th, 2008 at 1:13 pm PST) by Leon

Some customers in the frisky cluster (you can tell by going to the panel, clicking on “Account Status” in the upper, right hand corner, and looking where it says “Your Email Server” may be having problems getting to their sites. We are currently patching/upgrading the filer and it should be working again in 15-20 minutes. Sorry for the unplanned outage!

Affected machines: barracuda carrera chevelle corvette dart delorean esprit fairlane frisky-vserver10 geep hornet jetta knocker model-t montecarlo olds passat rambler riviera scion skylark tempest touareg

Update: Okay! Things are getting back to normal. Sorry about that. Please let your happy DH tech support person know if you’re still having problems!


8:10 pm

File Server Upgrades 2008-08-27

Posted (August 27th, 2008 at 8:10 pm PST) by Patrick

We’re almost there! Over half of the the file server upgrades are complete. Including tonight, we should only have three more nights of fun. On the slate for this evening, zoot at 10pm followed by bumper followed by kermit if time permits.

Check back here for updates!

Update @23:06

Zoot is progressing nicely. It should be back up and running in about an hour and a half. We’re starting on bumper now.

Update @00:09

Zoot is back up and serving files. Progress on bumper is moving along as expected. We’ll start kermit within the next 30 minutes.

Update @00:58

Bumper is back up and running. We didn’t start kermit as planned. That will happen now.

Update @04:17

Whew! After some issues with it’s boot archive, kermit is finally upgraded/patched/running! That’ll do it for tonight. Fozzie, thog, and gladys tonight will round out the upgrades for the next couple of weeks.


8:02 pm

File Server Upgrades 2008-08-26

Posted (August 26th, 2008 at 8:02 pm PST) by Patrick

Tonight we will be shooting for three! First up is sweetums at 10pm. Following will be clean up from last week (gonzo). If time permits, we will round it out with waldorf. Check back for updates.

Update @23:23

Unfortunately sweetums is doing the same thing janice was last week, and cannot be upgraded at this time. It is back up and serving files. The gonzo upgrade is starting now.

Update @00:29

Gonzo is progressing nicely and should be back and upgraded in about an hour. The problem with sweetums has also been resolved, so we’ll be starting that back up now.

Update @02:25

Gonzo has been up for about an hour now and sweetums just came back up! Both are serving files properly. The waldorf upgrade will start in about 10 minutes.

Update @04:47

Waldorf is back up and serving files! If you have any problems at all, please contact support.


2:23 pm

York is in trouble

Posted (August 26th, 2008 at 2:23 pm PST) by andrea

We have been seeing some problems with the York webserver earlier today. Reboots fixed but the problems came back. An admin is looking into it now, so we can find the problem and take care of it for good.

Very sorry about the inconvenience this is causing and delay in posting about this; we were under the impression that the problem was solved earlier. Please check back here for updates.

UPDATE: 2:47 pm – York has come back online and all apache web services are now operational. The low load is holding as well. If you’re still having problems with your sites, it may be unrelated, so please contact support with all details you have.

UPDATE: 8/27/08 9am – it appears that the changes we made on the server to fix the apache web services have been reverted. The problem is being escalated as it needs further looking into. We apologize for the continued inconvenience, please check back for updates.

UPDATE 8/27/08 2:30pm – we have found the problem and removed it. The server is good to go! Sorry about the delay on the fix. If you’re still having problems with your sites, it may be unrelated, so please contact support with all details you have.


10:37 am

Pegasus is being moved to new hardware

Posted (August 26th, 2008 at 10:37 am PST) by andrea

The server Pegasus was having troubles all morning. After trying the usual solutions, our admins are now resorting to move the server to new hardware. This process shouldn’t take more than an hour, so you can expect your sites to be back up by then. We don’t anticipate any extra issues with this, but please check back for further updates.

We apologize for all inconvenience this is causing!

UPDATE 11:01am – Pegasus is up again, sooner than we anticipated :) Please check your sites and let support know if you’re still having problems. It may be unrelated, so please include as many details as you can.


8:04 pm

File Server Upgrades 2008-08-25

Posted (August 25th, 2008 at 8:04 pm PST) by Patrick

Tonight we will be upgrading rizzo and bambi! The upgrade of rizzo will begin around 10pm PST. Bambi will follow as soon as rizzo has completed. Check back here for updates!

Update @02:15
rizzo is upgraded. Unfortunately we got a late start with this one. The upgrade process didn’t start until around midnight. We’ll be upgrading bambi in about 15 minutes.

Update @05:31
bambi is back up and serving files. The upgrade went well, but unfortunately it’s mirror broke, which means it will need to go down again at some point to be fixed. This will likely happen tonight. There will be a new status post with more information when appropriate.


5:32 pm

Oj is all squeezed out!

Posted (August 23rd, 2008 at 5:32 pm PST) by Leon

The webserver oj has been having some trouble lately which we believe is hardware related. We are in the process of moving it to new hardware and it should be back up soon.

Update 7:05pm PDT: Oj is now back up on its new hardware and is looking happy! If you’re still experiencing any issues, please contact support!


 
© 1996-2008, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
3:10 pm

PHP4 will no longer be available for *new* domains starting 08/26/08

Posted (August 22nd, 2008 at 3:10 pm PST) by jeremy

That’s right, since PHP4 has been officially discontinued, we’re taking our first steps in discontinuing our own support. If you currently have any domains using PHP4 (in either mod_php or CGI mode), they’ll continue to work for the foreseeable future. The only change occurring is that you will no longer be able to set up a new domain as PHP4 or convert an existing PHP5 domain to use PHP4. The few remaining one-click packages we had that required PHP4 are being upgraded simultaneously so that we’ll have no one-click packages that won’t work with new domains. If you’re in a position to do so, this might be a good time to start working on getting your websites working with PHP5 so that the later transition is easier. In most cases, the switch should be trivial.


11:16 pm

File Server Upgrades 2008-08-22

Posted (August 21st, 2008 at 11:16 pm PST) by kitchen

Tonight we’re going to be doing janice and gonzo again, starting with janice at about midnight pacific. We believe we have resolved the issues we had last night with the upgrades and things should go pretty smoothly. Check back here for updates as things progress!

Update: 12:09am pacific: We are getting started on janice right now.
Update: 3:09am pacific: Some additional problems were encountered with the upgrade of janice, and we are again aborting the upgrade. We will be rescheduling this upgrade for a later date. We greatly apologize for any inconvenience this has caused. We will begin the upgrade on gonzo at about 4:30am pacific and we do not anticipate the same problems we had with janice, so it should go very smoothly.
Update: 4:32am pacific: We are cancelling the upgrade for gonzo for tonight. It’s getting too late now, and east coasters will be getting up soon and wanting to check their email and whatnot. As of right now everything’s up and running fine on both of these systems, so if you’re having issues, please contact support!


10:00 pm

ns2.dreamhost.com maintenance 08/21/2008

Posted (August 21st, 2008 at 10:00 pm PST) by jordan

Tomorrow we will be upgrading ns2.dreamhost.com to apply some security patches. The actual downtime should not last longer then 5 minutes but during this period you may notice site slowness due to dns lookups failing at ns2.dreamhost.com. The upgrade is planned to start at 10:00PM PDT and should be finished around 11:00PM PDT.

UPDATE 10:00PM PDT –
The upgrade is starting now, this post will be updated then it has completed.

UPDATE 10:35PM PDT
We have finished with the security upgrade of ns2.dreamhost.com . If you are experiencing problems with ns2 please contact technical support.


11:22 am

FastCGI changes for some more users today (08/21/08)

Posted (August 21st, 2008 at 11:22 am PST) by jeremy

We had very few directly-related issues with the previous batch so we’ll be doing another group of servers shortly. It’s the exact same change as before (changing from mod_fastcgi to mod_fcgid) and should go very smoothly. The list of affected servers is:

alexander arkanoid attila baht bam bob bomberman caesar centipede contra cornelius cortes crack cruisin cssmania deblume defender digdug dinar dinero dollar donkeykong ellipsis euro fidget forint franc gage galaga gap go grant guilder happy herod ikari ike io ivan jezebel joust karnov katamari khan klax korben kroner leeloo lei limbo-randy limbo-randy2 lira lucky mark marvin morgan napoleon nexus nimitz orbital pacman paperboy patton peso pizarro pound qbert quid rhod ruble rupee rygar scagnetti schilling scipio shekel slaughter slimy tetris ugh ultima venice vex vito wasabi yen yoda zorg


7:06 am

Rock Smash

Posted (August 21st, 2008 at 7:06 am PST) by Sandon

The web-server named ‘rock’ has failed. All attempts to getting it back up have also failed; thus, I am failing it over to new hardware right now. It should be back up within an hour. You can check your web-server by clicking the “Account Status” link in the panel. If your web-server is not ‘rock’ then this outage does not apply to you.

Update: 07:40 AM PDT: The machine which hosted ‘rock’ has been failed over to new hardware and all services should be restored. Please contact our Happy DreamHost Support Team if you are still having issues with your site and you are on ‘rock’


10:10 pm

File Server Upgrades 2008-08-21

Posted (August 20th, 2008 at 10:10 pm PST) by kitchen

And the final 2 file servers for this week are going to get upgraded tonight:

janice
gonzo

We’ll be starting with janice in about an hour, and proceeding with gonzo after janice is completed.

As always, we will post updates here as things progress!

Update: 11:15pm pacific: we are starting on janice right now.
Update: 12:56am pacific: due to some problems with the upgrade process, we are aborting the upgrade on janice for the time being. We will likely finish this up tomorrow night. Sorry for any inconvenience :( Gonzo we will be starting around 4am pacific.
Update: 3:48am pacific: We are starting the upgrade process on gonzo now.
Update: 4:35am pacific: Gonzo had similar issues to janice during the upgrade process and we are aborting its upgrade as well. We will likely do these tomorrow night. Very sorry for any inconvenience.

Update: 5:15 AM. All services should be restored. Please contact the Happy DreamHost Support Team if you are still having any issues accessing your sites.


11:15 pm

File Server Upgrades 2008-08-20

Posted (August 19th, 2008 at 11:15 pm PST) by kitchen

Tonight’s round of upgrades include the following servers:

boppity
scooter

We will be starting with boppity tonight in about 20 minutes. We will post updates as things progress!

Update: 1:58am pacific: boppity is upgraded and we’re currently working on taking care of straggling issues before we get started on scooter.

Update: 2:39am pacific: ok, boppity has the all clear. We’ll be starting on scooter momentarily.

Update: 6:32am pacific: scooter’s upgrade is complete and things should be back to normal. If you’re experiencing any issues, please contact support!


11:25 pm

File Server Upgrades 2008-08-19

Posted (August 18th, 2008 at 11:25 pm PST) by kitchen

As mentioned last week, we will be performing upgrades to some of our file servers over the next few weeks. Tonight’s file server is animal (and hopefully gonzo afterward)

We will be starting the upgrade shortly. We will post updates as things progress!

Update: 3:09am: the upgrade has been completed and the file server is back online. We are currently working on cleaning everything up and getting things completely back to normal, we will post an all clear shortly!

Update: 3:45am: things should be back to normal now. If you’re still having issues, please contact support!


11:09 am

Redbull not so energetic, getting new hardware

Posted (August 18th, 2008 at 11:09 am PST) by Leon

Unfortunately, redbull has crapped out on us today. I’m moving it over to new hardware and it should be back in about an hour.

Update 12:35: Looks good! Let your helpful DH tech support person know if you’re still having trouble!


6:53 pm

Garth getting new hardware

Posted (August 16th, 2008 at 6:53 pm PST) by kitchen

Garth has experienced a hard drive failure and is currently being moved to new hardware. We expect to have everything back up within the next 30-45 minutes.

We will post an update again here when things are finished!

Update: 11:20pm: oops! this has been done for quite some time now, actually. Sorry for failing to update.

If you have any issues, please contact support!


 
© 1996-2008, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
3:45 am

Filer fun!

Posted (August 16th, 2008 at 3:45 am PST) by DreamHost

One of our file servers in the “frisky” cluster is experiencing some trouble. Our admins are en route to the datacenter now to get it back online ASAP. Updates to follow.

Update: 7:06AM Everything is back online. Sorry for the troubles.


4:12 pm

Network/Router Problems

Posted (August 15th, 2008 at 4:12 pm PST) by Ralph

It seems our network upgrade did not go as smoothly as we had hoped. We are currently experiencing network problems at our router which are causing downtime and slowness for all services that we offer. A team of administrators are investigating this issue. We will update this post as we have more information for you.

Update 5:30PM PST August 15: We have removed the bandwidth provider that appears to be causing all the problems. Sites should be up now, you should be able to access the panel, and services should be faster than before. We are leaving this status post as unresolved until we are positive that everything is better.

Update 5:58PM PST August 15: We have added the bandwidth provider back in and it appeared to cause some problems again initially. It is still up and things seem to be improving. Still not resolving this issue yet and reading your comments as well as contacting customers with support messages to see how things are from your end.

Update 6:04pm August 18: Things appear to be working properly now.


11:30 am

Central Services Outage

Posted (August 15th, 2008 at 11:30 am PST) by kitchen

One of our central database servers has kernel panic’d and is currently being rebooted. For the mean time, most central services are offline (web panel)

We have our mysql gurus working on it and we expect to have things back up and running shortly.

UPDATE 1:56 PM: Okay this has been resolved and our panel is up again. This would not have caused domain downtime, so if you are having issue it is not related and you will need to contact support.


10:25 am

Frisky File Server Outage

Posted (August 14th, 2008 at 10:25 am PST) by brians

Our Admins had to replace several drives on one of the fileservers that your machines are mounted to. The drives are rebuilding at the moment. Therefor performance isn’t quite at 100% at the moment. This will explain any problems that you are encountering when attempting to access your site, email, etc. Unfortunately, we do not have an ETA as to when this issue will be resolved. We sincerely apologize for any inconvenience this has caused and appreciate your continued patience. Please check back for future updates.

UPDATE: RAID reconstruction is at 52% and counting. This does not have any affect on your service, other than the slight delay that you may be experiencing. Meaning, you do not have to worry about data loss and such. We will make another posts once it complete. Again, we apologize for the inconvenience this has caused.

Update: reconstruction is complete and all services should have returned to normal now.


4:03 pm

Apache upgrades, No more Telnet, no more Frontpage Extensions!

Posted (August 13th, 2008 at 4:03 pm PST) by Patrick

The winds of change are blowing! Starting next week, we will begin to phase out telnet and Frontpage, as well as upgrade apache services to Apache2.2 (from Apache 1 and Apache 2).

If you are still using telnet to connect to a shell, you should immediately switch to SSH. For information on SSH and how to use it with DreamHost, see this wiki article.

As we posted back in June, Microsoft has discontinued Frontpage, which means we have no way to support its extensions in Apache 2.2.

– Whatever extensions you have will stop working.
– You will no longer be able to upload your site within Frontpage using Frontpage Extensions. You will need to use an FTP client instead. You can find information on FTP clients here. Note that Frontpage does have a built-in FTP client that you can use as well. Instructions on using Frontpage as a FTP client can be found here.

If you have any questions or concerns, please contact support.


2:59 pm

Joomla 1.5.6 Available!

Posted (August 13th, 2008 at 2:59 pm PST) by Jason C

A very serious security update for Joomla 1.5.x has been released as part of Joomla 1.5.6. Any and all customers using Joomla 1.5.x are encouraged to upgrade immediately either via the One-Click Install section of the web panel or by manually downloading and updating their install the old fashioned way.


3:54 pm

Network Maintenance retry! (Completed)

Posted (August 12th, 2008 at 3:54 pm PST) by DreamHost

On Thursday, August 14, 2008 at 10pm Pacific time we are going to retry turning up our remaining links with Global Crossing. They assure us the previous bugs have been fixed, and we shall try again. If all goes well, you shouldn’t notice any downtime. The window for this is 1 hour, though we should be finished in well under 30 minutes as the configuration work is completed.

http://www.dreamhoststatus.com/2008/07/30/network-maintenance-tonight-0730/

Update: 10:38pm Pacific – The links have been up for 30 minutes now, and everything appears stable. Testing customer websites from remote locations that route over both of our providers appears to be working fine. Please contact support if you are having any trouble!


3:50 pm

File server upgrades

Posted (August 12th, 2008 at 3:50 pm PST) by DreamHost

We will be continuing our file server upgrades to address a long standing SATA driver bug in Solaris 10. These will occur over the next three weeks, starting Monday, August 18, 2008 at 10pm Pacific time. We will be doing one to two per night, depending on any hitches which may present themselves along the way. Estimated downtime per file server is no longer than 4 hours.

These upgrades will affect your websites, email, ssh, and ftp access to your files if you are on one of these file servers. No email or data should be lost during this process. MySQL will not be affected.

Week of 8/18/2008
animal
gonzo
scooter
boppity
janice

Week of 8/25/2008
rizzo
bambi
sweetums
waldorf
zoot

Week of 9/1/2008
bumper
fozzie
thog
gladys
kermit


12:35 pm

Upgrading Ruby on Rails to version 2.1.0

Posted (August 12th, 2008 at 12:35 pm PST) by kitchen

Howdy folks!

We’re currently working on upgrading our rails packages from 2.0.2 to 2.1.0. We expect to have everything packaged and tested by the end of the week, with the new packages starting to roll out on next tuesday.

The best way to make sure this doesn’t break your existing rails applications is to go into your rails application’s base directory and run:

rake rails:freeze:gems

Also with this update we’ll be deprecating our support of Rails 1.2.6. If your application still relies upon Rails 1.2.6, you will either need to update your application for support of rails 2.1.0, or freeze your app to the tag rel_1-2-6, with the following command:

rake rails:freeze:edge TAG=rel_1-2-6

If you have any questions about either the upgrade to 2.1.0, or the deprecation of 1.2.6, please contact support, and we’ll be happy to answer you!


11:02 pm

Moving pegasus to new hardware

Posted (August 9th, 2008 at 11:02 pm PST) by DreamHost

The webserver pegasus is having some trouble, so I’m moving it to new hardware. This process should take about thirty minutes.

Update: This should be finished, please contact support if you have any sites hosted on pegasus that are not working.


 
© 1996-2008, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
3:36 am

Short MySQL downtime on 08/12

Posted (August 9th, 2008 at 3:36 am PST) by Sandon

The MySQL servers listed below will be going down shortly for maintenance at around 2 AM PDT on August 12th for maintenance. The maintenance should only cause about 10 minutes of down-time (total) but even if things go badly should not take more than 30 minutes to complete so there should not be more than 30 minutes of total downtime.

Here is the list of the MySQL servers:

aslan, burbank, cassandra, dandilo, figaro, garfield, guinness, hobbes, luna, meow, mimsey, natty, orion, roary, rumple, snowball, stimpy, toonce

Update: 02:25 AM PDT

All the servers have been back up for about 10 minutes now so the total downtime was about 15 minutes which was a bit more than expected but well below 30 minutes. If you are still having any problems connecting to MySQl please contact support.


10:08 am

Quid has faulty disk, failing over to new hardware.

Posted (August 5th, 2008 at 10:08 am PST) by Kyle

Quid started spewing disk errors to the kernel log a few moments ago and has continued to do so after a reboot. The server is being moved over to new hardware, estimated downtime is one hour.


8:54 pm

Junk Mail Currently Inaccessible

Posted (August 4th, 2008 at 8:54 pm PST) by Jason C

Due to a database error, any users trying to get at their Junk Mail settings right now will run into some problems.  Our admins are currently restoring the tables for all Junk Mail users and the system should be back up just as soon as that restore completes.  It might take a couple hours to run thru, but we’ll be sure to update this status blog post as the restore rolls along.

Update: This entry should have been updated when the restore completed — but that doesn’t appear to have been the case. Any users continuing to see issues with their Junk Mail settings and quarantine should contact Technical Support at once.


10:25 am

Last chance to donate to SPCALA / Communities in Schools

Posted (August 4th, 2008 at 10:25 am PST) by Josh

Yep, it’s about time we change those charities that you can donate to directly from your DreamHost bill and have it matched by us:

Go visit our panel and contribute today.. so far SPCA LA has received 17 pledges for a total of $632.03 and CIS has received 20 for $318.56.

MORE PLEASE!


2:14 am

A Spunky DreamHostPS host is getting new hardware

Posted (August 4th, 2008 at 2:14 am PST) by kitchen

One of our DreamHostPS host machines has been having some hardware difficulty this evening and we’re going to be replacing it. The affected DreamHostPS machines are:

beachhead darklandz ezasamblea mp sixside vahati

We expect to have this resolved within the next 2 hours. We apologize for any inconvenience and thank you for your patience.

Update: 3:31am pacific: As you may have noticed, your sites are still online. We were able to get the old hardware up and running while we prepared the new hardware for failover. We will have to take them down for a few more minutes in a little while, but for the time being they’re up again (until we either take them down for the failover, or the hardware dies again!)

Update: 4:00am pacific: We have taken the old hardware offline and are currently moving things over to the new hardware. The DreamhostPS guests will be coming back up shortly. We will update again when things are all clear!

Update: 4:18am pacific: the hardware failover has been completed and the guests are all up and running just fine! If you have any issues, please contact support!


8:59 am

WebMail Down

Posted (August 1st, 2008 at 8:59 am PST) by brians

We are fully aware of the authentication issues that you are experiencing when accessing your Mailbox via WebMail. There appears to be an issues with one of our central databases. I have an admin looking into the issue at the moment. We hope to have this issue resolved shortly. Sorry for any inconvenience this has caused. Please check back here for further updates.

–UPDATE–Fri Aug 1 10:51:59 PDT 2008 (17:51:59 GMT)
We’re still working to resolve the issues with webmail. Currently we recommend all webmail users give roundcube.dreamhost.com a try. Roundcube is our beta alternative to SquirrelMail. The initial login to roundcube just requires you to input your entire email address and then your password. Don’t be confused by the @dreamhost.com portion of the user login. As a matter of fact, pretend it’s not even there! Simply enter:

Username: user@domain.com
Password: yourpassword

Sorry for the inconvenience and we will have SquirrelMail working properly as soon as possible.

UDPATE: The problems with WebMail have been resolved. One of the WebMail machines was deleting sessions. The machine has been fixed. I’ve tested user accounts extensively as well. Again, we sincerely apologize for any inconvenience this may have caused. If you continue to have problems with webmail access, please submit a support ticket.


11:51 pm

Dalitz crashed, filesystem check

Posted (July 31st, 2008 at 11:51 pm PST) by DreamHost

The web server dalitz crashed a little bit ago, and has been checking its filesystem since. It should be back up inside of an hour. We apologize for this inconvenience, these filesystems take about that long to check.

Update 12:17AM Pacific (Friday): Looks like Dalitz has come back up and is running normally.


6:38 pm

Pligg 9.9.5 Available In One-Click Installer

Posted (July 31st, 2008 at 6:38 pm PST) by Jason C

A major security update has been released by the Pligg team today and it is suggested any and all customers who utilize our one-click install of Pligg to upgrade immediately.  That can be done from the One-Click Installs sub-section of our web control panel under the Goodies section.  If you happen to see any issues with the upgrade, please contact our support department right away.


5:42 pm

ns2.dreamhost.com nameserver issues

Posted (July 31st, 2008 at 5:42 pm PST) by danwy

We are currently having issues with our ns2.dreamhost.com server. It is not responding to queries and it may cause issues if you’re trying to point a new domain to our nameservers. After a quick check, we believe this may be due to a bad drive in the server. Our admin team is currently working on this situation and hope to have it resolved as soon as possible. We will update this post when we have more information or the issue is resolved. We’re sorry for any inconvenience this causes you.

Update: This issue has been resolved.  Basically, the nameserver had an issue with database replication and needed a bit of TLC from our admin team to get it working again.  It is now accepting queries and any issues caused by this should now be taken care of.


11:48 am

Capone is down

Posted (July 31st, 2008 at 11:48 am PST) by brians

This will explain any problems that you’re encountering when attempting to access your website, ftp server, etc. I have notified an admin and they are currently looking into this matter. We truly apologize for any inconvenience this has caused. Please check back here for future updates.

UPDATE: Capone is back online. Fscking a 4TB volume prolonged the reboot. Again, we apologize for the inconvenience the downtime caused and truly appreciate your patience. If you continue to experience problems, please submit a support request.


 
© 1996-2008, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
9:17 am

Moving Cheezit to new hardware

Posted (July 31st, 2008 at 9:17 am PST) by brians

Our Admins are diligently working towards getting the machine back on line. Sorry for the inconvenience this may have caused. Please check back here for future updates.

Cheezit is live on the new chassis, contact support if you have any lingering issues


5:31 am

Network blip for some customers

Posted (July 31st, 2008 at 5:31 am PST) by DreamHost

This is directly related to the network turn-up we did late last night referenced here:

http://www.dreamhoststatus.com/2008/07/30/network-maintenance-tonight-0730/

It appears that the IP access lists were not matching with GlobalCrossing and ourselves, causing a subset of our users whom routed in via GlobalCrossing to a specific subset of our IP’s to not be able to reach DreamHost. The GlobalCrossing link has been shut down until we can rectify this with their BGP team. All traffic is flowing over our other internet connection for now. We apologize for not catching this sooner, we simply did not see any support volume related until people began to wake up about 30 minutes ago!

Update (Minutes later!): I just got off the phone with GlobalCrossing and it seems that some interaction between our two Cisco routers was causing some form of CPU spike on both sides. We are both in the process of opening cases with Cisco support to figure out what was causing this problem. The plot thickens! We will be leaving these uplinks offline until this bug can be resolved.


5:22 pm

Spunky Network File Server Issues

Posted (July 30th, 2008 at 5:22 pm PST) by Jason C

Some customers on the spunky cluster might be seeing some outage issues. One of our network file servers is acting up and members of our admin team are scrambling to get this matter resolved quickly.

Possibly affected servers are as follows: admiralty aero alameda alondra archibald arrow atlantic avalon aviation babyruth barry bazooka beaudry bellflower betsybird blowpop bonbon bottlecap boyd broadway burton butler catalina centinela century cerritos chex cognac compton cookie cordial cupcake danielof dingdong doheny dorito drumstick dumdum edendale euclid fairfax fireballs fletcher foothill fundip garland garth garvey gobstopper goober good gower grand grappa gumdrop haven hillhurst hillmont hoover hope hops hyperion knocker lankershim larchmont lemonhead limbo-spunky4 linq locust look lucas ludlow lydia malt manchester mayo melrose milliken mills monroe munchies nilla ninja nix oakdale ocean olive ord overland pernod pico pikespeak plenty pocky poprock pretzel price pringle razzle ritz rockyroad rodeo roscoe rossmore rum sake saltine schnapps selma skittle skybar slauson snicker soto sourpatch spunky-vserver03 spunky-vserver16 spunky-vserver19 spunky-vserver27 spunky-vserver39 starburst sunset taffy tampa temple tequila topanga twix vanowen warhead washington wayne western whittier whopper wilshire york

We’ll keep you updated as the matter persists.

Update: Everything is happy again! It appears that the file server temporarily lost its routes. Our admin team is cleaning up a few straggling services and sites should be up again for the most part. If you happen to notice any further problems, please contact Technical Support immediately and we’ll gladly help you out.

Update: Unfortunately we have to repair a disk on the same file server which we are doing now – we’re hoping this will take no longer than an hour but we’ll update this post again when it is resolved.

Update: Good news – the repair has already completed so services are coming back up. In addition to that the admin team is offloading this file server so that it will no longer cause problems.


10:53 am

Network maintenance tonight (07/30)

Posted (July 30th, 2008 at 10:53 am PST) by jeremy

Tonight, around 10pm, we’ll be plugging in 5 new network links to expand our bandwidth by 50%. We don’t expect any downtime to be incurred but we’ll have our Cisco consultant on-site in case anything happens.

Update 11:53pm Pacific: We turned on the links, and we seem to be cooking along. The remote router hiccuped once, but we have been stable for over an hour now.

Update 05:41am Pacific: http://www.dreamhoststatus.com/2008/07/31/network-blip-for-some-customers/ Seems it did not go as well as it first appeared.


3:30 pm

FrontPage Extensions will no longer be supported in the next few months.

Posted (July 29th, 2008 at 3:30 pm PST) by Brian@dreamhost

It’s been a long, tumultuous relationship; we’ve laughed, we’ve cried, we’ve had many ups and downs, however, it’s time to part ways with Microsoft’s FrontPage web editor.

In the next few months, we’ll be moving over to a new version of Debian Linux, and this means we have to leave some software behind. We’ve done our best to support Frontpage as long as possible, even long after Microsoft has left it behind:

http://office.microsoft.com/en-us/frontpage/default.aspx

“What happened to FrontPage?

After nine years of being an award-winning Web authoring tool, FrontPage will be discontinued in late 2006. We will continue to serve the diverse needs of our existing FrontPage customers with the introduction of two brand-new application building and Web authoring tools using the latest technologies: Office SharePoint Designer 2007 for the enterprise information workers and Expression Web for the professional Web designer.”

We know this will cause some problems for users, and we do apologize for the hassle this may cause. We can say that FrontPage will still continue to work as a standalone FTP program, and you can still use it to design pages, we’re specifically talking about the “extensions” that FrontPage uses.

Again, we’re sorry for any trouble this causes, and hope the transition is smooth for everyone involved.


11:56 am

Duck and cover!

Posted (July 29th, 2008 at 11:56 am PST) by Leon

Yep, we just had a decent sized earthquake in the greater LA area, but everything looks fine so far. We’ve done a quick inventory of our machines down in our datacenter and everything appears to be okay.

We’ll update this post if any issues manage to come up due to the earthquake.


11:16 am

Changes to Junkmail Quarantine

Posted (July 29th, 2008 at 11:16 am PST) by jordan

If you have Dreamhost Junkmail enabled for your domain, the process of checking your junkmail quarantine has changed. Now instead of using webmail for accessing the quarantine you will now have to login to http://mailboxes.dreamhost.com with your email address and password. After this you will see a link to access the new Junkmail Quarantine. One of the new features of this upgrade is the ability to change junkmail users settings domain-wide and there will be plenty more to come. If you have any questions / problems using the new interface please feel free to contact support.

The old Junkmail interface is disabled as of now so we can sync the new system with the old. This will cause your to be unable to access junkmail for a short period of time. We expect to have the new quarantine system live by 1:00PM PDT and will resolve this post when it is complete.

UPDATE 1:29PM PDT

Unfortunately the syncing of the old and new system is taking a bit longer then we expected. Once the sync is complete the new system will be activated and more information will be added to this post.

UPDATE 7:00PM PDT

The new junkmail quarantine is live. We do apologize for the short notice of this system change but we assure you this change is for the best and is laying the ground work for many new and exciting junkmail features. You can now make changes to your junkmail users settings domain-wide via the Mail->Junkmail tab of the Dreamhost Panel and there will be more features coming in the next few weeks. Due to the amount of comments and emails received about the system change , we will be looking into giving users the option to have their junkmail quarantine show up as an IMAP folder and also integrating it into the various webmail clients we provide. If you experience any troubles with the new junkmail system please contact support and we will be happy to assist.

UPDATE 7:19PDT
We are continuing to work on issues with the new junkmail system, however the issues that have so far been seen with custom policies and forward addresses should now be resolved. Please contact support if you continue to experience problems with the new junkmail system.


8:02 am

Mail Delays for Postal cluster.

Posted (July 29th, 2008 at 8:02 am PST) by mir

This weekend we experienced a spam run on one of our clusters. Our admin team was quick to shut it down but it is still resulting in some delay of mail deliver as the MX servers get caught up. We’re keeping an eye on it and mail should not be lost at all but you will see 1-2 day old messages showing up over the next couple of days. We apologize for the inconvenience (spam is so annoying!).

Just a note, to check if you are on the effected cluster just open your Domains >> Manage section of the panel here:

https://panel.dreamhost.com/index.cgi?tree=domain.manage&

and click the DNS link for any domain that has mail hosted here though us (the MX records will show if you are on the postal cluster or not).

Update: While this issue is resolved and the queues on postal mail machines have leveled out, you will probably see messages that were delayed previously if you hadn’t logged in for a while.  At last glance tho, it looks like things are pretty much caught up now.  As always, if you happen to notice lingering issues, please contact Technical Support.


12:28 pm

FastCGI changes for some users on Monday

Posted (July 25th, 2008 at 12:28 pm PST) by jeremy

Due to very popular demand, we’ll be testing out a switch from mod_fastcgi to mod_fcgid for a small subset of users on Monday. This will _only_ affect those of you on the following machines who make use of our FastCGI support.

Affected machines:
abe annie apu arnie bart bitters brandine brawndo brockman burns carl chalmers cheeky clancy cletus dawber dib drederick duffman flanders gaz gimli gir glass gloop gotcha harpo hibbert homer jasper jebediah jimbo jubjub kearny krabappel krusty larry lenny lisa lovejoy manjula marge mccallister mcclure millhouse mmhper mona monkey ned nelson otto patty pogo quimby raiden ralphie salt sanjay seamus skinner smithers tak wiggum willie yerba zim


9:19 am

Hardware problems on Stewarts

Posted (July 25th, 2008 at 9:19 am PST) by Kyle

The network interface which is used for filer traffic is flapping like crazy. Numerous attempts to reconfigure and restart networking have proved unsuccessful. The machine is being moved to new hardware, estimated downtime is 1 hour.

Edit: Taking a bit longer than anticipated for some configurations to be pushed through, should be up shortly however

The pain is over, you can all thank the nic manufacturer that starts with B


 
© 1996-2008, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
11:23 am

Emergency file server OS upgrade (Canceled)

Posted (July 23rd, 2008 at 11:23 am PST) by DreamHost

The file server “scooter” is going to undergo an emergency OS upgrade to prevent a bug in its disk drivers from corrupting data on disk, and otherwise causing it to crash. This is unfortunately estimated to go all day. We have moved this upgrade up to today due to the problems we have had getting another file server back online. We shall be offloading all of the data on to the newly upgrade file server once it is complete:

Edit: Nevermind! We realized we have an unused one we are going to deploy and use for both purposes stated above. scooter never went offline.


11:39 am

WebMail Down

Posted (July 21st, 2008 at 11:39 am PST) by brians

We experienced an outage on the central database the handles WebMail. Downtime lasted 30 – 40 minutes. Our Admins have restarted the database server. This appears to have resolved the problem. We’ll continue to actively monitor the stability. Sorry for any inconvenience this may have caused. If you continue to encounter problems with Webmail access, please submit a support request.


11:17 am

Lovejoy and Sanjay File Server Issues

Posted (July 21st, 2008 at 11:17 am PST) by Justin@DH

A file server that hosts files for a small number of customers, mostly on the lovejoy and sanjay web servers, is having issues at the moment. Our admins are working to have the file server check its system and disks, which must happen before it can go live again. This is causing website and email downtime for the customers on these two web servers. We are working to have this fixed as soon as possible, when we have further information we will update this post.

Update – 03:45 PM PST: Recovery efforts are still moving forward. At this point it will take at least 16 more hours for disk checks to complete. And that’s in a perfect world. Please accept our most humble apologies for any and all problems this is causing the affected users of lovejoy and sanjay.

Update: 1:47 pm Pacific: The file server is back up and running. No file should have been lost, please let tech support know if anything seems amiss! We’re sorry it took so long to get up, once the filesystem checks finished the machine did not want to boot back up into its operating system.

Update – July 22nd 04:59 PM PST: Unfortunately the problems with the file server returned again. We are starting up another file and disk check on it. We will be watching the check closely to make sure that we do not miss any problems that we possibly missed. Our apologies and we will get things up for you as soon as we can.

Update – July 23rd 11:27 AM PST: The best update for this specific issue is the new post regarding the scooter file servers emergency OS upgrade:

http://www.dreamhoststatus.com/2008/07/23/emergency-file-server-os-upgrade/ (Canceled)

We had continued issues with the file server used for lovejoy and sanjay so we have decided to get your files moved to new hardware. As explained above the upgrade is expected to take a day, then after that is done we will start on the moves. We will limit downtime as much as possible.

Update – July 23rd 12:58 PM PST: As you may have noticed the plan with OS upgrade above was canceled. We have decided to add a new unused file server to our data center and move the data to that one instead. This will limit the number of people experiencing downtime, but will keep pretty much the same time line for those of you on Sanjay and Lovejoy. So the new hardware does need to be upgraded still, then we will start the data moves.

Update – July 24 7:20 PM PST: Data moves should be starting tomorrow to the new file server. We will do the moves on a user by user basis, so once the data copies over your user will be restored. If we have a change in plans we will update this post again. Sorry for the continued issues for those of you with users on this file server, we will make sure to get you a credit for the downtime.

Update – July 25 5:01 PM PST: The data moves to the new file server have been running since around noon. We are trying to get a good estimate for all of you affected. Unfortunately the estimate for the move is next Friday. If you have email addresses that are vital set up a new username and set the mailbox to that user. The user will not be on the problem file server. Also if you have a domain you need up and you have the files locally, you can switch the user for the domain and upload your files to get your domain running. I know this work around does not work for everyone but some users were happy with that solution while we finish dealing with this issue. Our sincere apologies for the downtime this has caused.

Update – July 31 11:47 AM PST: Unfortunately nothing new to report. Moves are still running successfully at this time. A lot of you have open support tickets and I will be getting to you one at a time to see what options I can give you. In some cases your data has moved and I have to run some manual updates to get things working for you. Hang in there. I’d have to say the estimate for all of the data completely moving seems to be taking longer than expected. Each user on this file server have tremendous disk usage so the moves take hours to complete for one user. If you have an open ticket I will do my best to get to you today.

Update – August 1 10:50 AM PST: So far the move has completed 36 moves with around 59 more users to go. We did unfortunately have a hiccup during the previous move which set up back about 24 hours. Now the moves are running well. Since each user account takes a various amount of time it will be difficult to give you an accurate ETA. What we’ll do is post how many users are left to move at least twice a day.

Update – August 1 6:20 PM PST: 51 users left to go. We can mount up some month old backups for you if you really need your files. If you are in a situation where you just want access to your files we’ll set up a user.backup user for you which will have the same password. The data will not be live, but will be accessible. If you would like this please send a support message to dhstatus@dreamhost.com. You must send from the primary contact email address on your account.

Update – August 2 2:05 PM PST: 50 users left to go still. A really large user is moving right now as well as two others.

Update – August 3 2:20 PM PST: 45 users left to move.

Update – August 5 11:04 AM PST: 36 users left to move. Sorry for the missed updates yesterday.

Update – August 5 7:54 PM PST: 34 users left to move. There are currently 4 moves running.

Update – August 6 1:11 PM PST: 33 users left to move. 4 moves are running.

Update – August 6 7:05 PM PST: 32 users left to move. Some of you have reported that the email work around is still causing a bounce even though you are receiving mail at the new location. You will actually just need to set the address as a forward only. Once the move is completed you can switch back to a full mailbox. We have 4 moves running.

Update – August 7 9:24 AM PST: 27 users left to move.

Update – August 8 9:56 AM PST: 23 users left to move.

Update – August 8 6:07 PM PST: 22 users left to move. Two moves are running currently. We have been able to get the sites working that have moved. Sorry this move is taking so long, we have the machines plugged directly into each other and the moves are going much slower than we typically see. It is possible that the problems that caused the stability issues are somehow causing the performance issues with the moves. We’ll keep going though. Our sincere apologies and credits will be issued once this issue is finally resolved.

Update – August 9 9:42 AM PST: 19 users left.

Update – August 11 11:16 AM PST: 1 user left to move. User actually already moved but there was a problem so we have to remove them in order to ensure that everything is okay. Once that user moves we will need to fix some mail mounts then issue credits. At that time the problem will be resolved. If you do notice problems right now please contact support for assistance.

Update – August 11 6:28 PM PST: Okay we have finished everything and it should all be working. If you notice any further problems please contact ralph@dreamhost.com. Make sure you write in from your accounts contact address. He’ll fix anything left over. Credits will show up on your accounts by this time tomorrow.

Update – August 12 10:36 AM PST: Thank you to those of you who have contacted me directly. We are still having some problems with mounting some user directories for mail only. I am working with our administrators on this issue with the examples you gave me. I will update as soon as we have this resolved or have more information.

Update – August 12 5:36 PM PST: We feel confident that this problem is resolved now. All email is being reported as working now and credits have been issued. You can see the credit from the Manage Account->View Invoices area of the DreamHost Webpanel. It is an Account Service Credit of $119.40 paid on 8-12-2008. Sorry this took so long to resolve and if you have any further problems just let me, ralph@dreamhost.com, know.


6:38 pm

MySQL Downtime

Posted (July 18th, 2008 at 6:38 pm PST) by Ralph

We are currently noticing some downtime with MySQL servers. Our administration team is looking at the issue right now. We are experiencing heavy packet loss on one of our MySQL racks in the data center. We will update this post with more information as soon as we can.

Update 7:24pm – We have fixed the issue. It appears that some network monitoring software we were using got a bit out of control and clogged the network. We have disabled it for now and service has returned. Sorry about that.


5:33 pm

Hoover sucks! New hardware!

Posted (July 18th, 2008 at 5:33 pm PST) by Leon

I couldn’t resist with that headline. Hoover’s been up and down all day today, so we’re moving it to new hardware. Should be back in about an hour. Sorry about that (and bad jokes)!

Update 6:35: Okay, 2 minutes over an hour. My bad. should be good to go now. Please notify your friendly DH support person if you have any other problems!


12:45 pm

Scion hit a semi, moving to new hardware.

Posted (July 18th, 2008 at 12:45 pm PST) by Kyle

Scion was powercycled to recover from the filer issue on frisky and didn’t come back. Currently in the process of being moved to new hardware.


11:45 am

Esprit is dead, moving to new hardware

Posted (July 18th, 2008 at 11:45 am PST) by Leon

D’oh!! Esprit’s not coming back from a reboot. I’m moving it over to new hardware right now, should be back in about an hour. Sorry about that!

Update 12:59pm: Okay, back to normal! I mean that in a good way. If you’re still having trouble, let our pals in tech support know!


9:59 am

Frisky Cluster Downtime

Posted (July 18th, 2008 at 9:59 am PST) by brians

We are currently experiencing on outage on one of the fileservers that your machines are mounted to. This will explain the connection issues that you are encountering when attempting to access your website, email, etc. Our Admins are currently working to resolve the issue. We sincerely apologize for any inconvenience this may have caused. Please check back here for future updates.

UPDATE: Had to rebuild a few drives. The fileserver is now fully functional. You should not experience any further issues with website, email, etc access. Again, we apologize for the inconvenience the downtime may have caused.


10:14 am

Spacey filer failure (Update!)

Posted (July 17th, 2008 at 10:14 am PST) by DreamHost

One of the filers for the Spacey cluster has suffered a hardware failure. Technicians have been dispatched to this site. Estimated downtime is one hour for affected customers. This includes all websites and email hosted on this filer, but not MySQL.

Edit: 10:26am Pacific It looks like a hard drive failed and was causing errors on the filer which prevented it from properly functioning. Remote hands have removed the faulty disk and the filer appears to be running normally. Our technicians are still en route with spare parts just in case! We are rebooting the affected servers now, and systems should be nominal in about 15 minutes.

Everything seems to be back up and running! Please contact support if you are still having problems.


7:19 am

Packetloss for some Dedicated customer servers.

Posted (July 17th, 2008 at 7:19 am PST) by Sandon

It looks like one of the switches which has the following dedicated servers hooked up to it:

saints nassweb bmw gafc dante blazer lifetimesvoice natcurl musicbox settlements meep perfecto peppermint naish motmot extensionmedia tekhost thechange verne sitepartner inaux calmini aeroquest1 cowboy argtracker lowest-airfare mqseries yaleweb auction evolution01 killeroldies singe

Is suffering from major packetloss (30-50%) which is causing extremely slow access to these servers. It should eventually serve up the data (because of the way TCP/IP works) but the symptoms would be extremely slow network downloads/page loads due to droped packets and ACKs (which have to be re-transmitted). We are aware of the issue and working on it. More update to come when they become available.

Update: We replaced the switch in question and things are working properly now. I apologize for the outage, if you dedicated server is still having any problems please contact support.


 
© 1996-2008, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
2:07 pm

Passat has crashed, moving to new hardware

Posted (July 16th, 2008 at 2:07 pm PST) by Leon

It looks like passat’s not coming back after a reboot. It should be back on new hardware in about 1 hour!

Update 3:32: Sorry for the long delay, that was no fun! Should be back up now, though.


4:36 pm

Brawndo Moving to New Hardware!

Posted (July 15th, 2008 at 4:36 pm PST) by Leon

The Brawndo web server had its NIC card fail, so we’re moving it to a new home. We expect to have everything back up shortly!

Update 4:50PM PDT: The server is now on new hardware and all services appear to be back online. If you’re still having trouble, contact support!

Thanks!

– The Happy DreamHost Server Failover Team


3:22 pm

Dali’s gone all goofy, moving it to new hardware

Posted (July 15th, 2008 at 3:22 pm PST) by Leon

Moving dali over to new hardware, should be back in about an hour. Sorry for the downtime!

Update: Sorry about the delay, we’re almost finished with Dali.

Update: Ok, Dali should be all up and working please contact support if you have any sites on it that aren’t working properly.


2:47 pm

Archibald Moving to New Hardware

Posted (July 15th, 2008 at 2:47 pm PST) by Leon

The Archibald server is having troubles on boot, so we’re moving it to new hardware. We should have everything back up shortly!

Upate 4:25PM PDT: Archibald is now back on new hardware and everything should be good to go. If you’re still experiencing issues, please contact support.


1:38 pm

Central Database Outage

Posted (July 15th, 2008 at 1:38 pm PST) by mir

Some bad queries have locked up our central database server which is effecting the panel function as well as several of our own internal systems. We are in the process of cleaning this up and restarting the database server but it takes time for everything to come back up. We expect this to be resolved by 3 PM, PST at the latest but will update if we receive information to the contrary. We appreciate your patience with this issue and ask that you check here for further details rather than attempting to contact support as we’ll have no way to load your tickets until after this is resolved anyways.

Update: The admin team was able to resolve this much faster than anticipated and it is working again with minimal downtime.


9:19 pm

WordPress 2.6 Upgrade Available

Posted (July 14th, 2008 at 9:19 pm PST) by Jason C

WordPress 2.6 is now available in the advanced section of our one-click install panel for your bleeding edge upgrading pleasure.  For more information on this version of WordPress and the added features, please visit the WordPress blog.


8:48 pm

Forwarding to Gmail Delayed

Posted (July 10th, 2008 at 8:48 pm PST) by Jason C

Customers are reporting severe delays with messages that should be forwarded to Gmail and Google Apps hosted domains.  It appears direct mail from our customers is coming thru just fine to Gmail addresses — but forwards are seeing delays of several hours.  If this sounds familiar, it is because it happened about 8 months back.  This means that we have a pretty solid idea of what is going on this time around tho.

Members of our admin team have been in touch with the Gmail team at Google and are working together on a resolution.  More information on this matter will be added onto this entry as new details bubble up to the surface.

Update: This issue has been resolved.

Update: To elaborate on the resolution, the fine folks at GMail found a bug which was causing this to happen, has fixed the bug, and is setting up some monitoring to catch this bug in the future. Much thanks to the GMail team for the quick resolution!

P.S. Forwarding-only accounts where important mail is expected over the next 24 to 48 hours should add a mailbox hosted here to guarantee mail delivery for the time being.  Hopefully things won’t take that long — since the matter is in Google’s hands, we don’t quite have an ETA — but it’s always best to plan for the worst case scenario.  If you need assistance with doing this, please contact the support department.


2:04 pm

Reminder: Network maintenance tomorrow night! (Updated)

Posted (July 10th, 2008 at 2:04 pm PST) by DreamHost

http://www.dreamhoststatus.com/2008/06/18/network-outage-july-11/

This is a reminder about the network maintenance we will be performing July 11, 2008 starting at 10pm Pacific time. All of DreamHost will be inaccessible during this downtime, including email, websites, jabber, streaming, mysql, etc. No data will be lost during this time, and email will be deferred until the network is back online and delivered over the course of the night by the sending servers.

Update 7/12/08 12:32am PDT:
The network was back up briefly before being taken down by an unexpected hardware fault on one of the central routers. It’s currently coming back up and with any luck, things will be returning to normal shortly.

Update 7/12/08 2:58am PDT:
Everything’s been back up for a while now and we’ve been monitoring everything very closely. From what we can tell, the previous issues are all gone and everything’s running smoothly. We’ll continue to monitor the network very closely over the next 24 hours to be absolutely certain no strangeness pops up.


3:03 pm

Server “laurel” being moved to new hardware.

Posted (July 7th, 2008 at 3:03 pm PST) by Leon

It appears that one of the NICs on laurel died today, unfortunately. We’re moving it to new hardware and it should be back within an hour or so. Sorry for the inconvenience!

Update: Whoops! Forgot to update you guys, but laurel’s back in bidniz. Please write to support if you have any lingering issues.


11:19 am

intermittent webmail issues

Posted (July 7th, 2008 at 11:19 am PST) by danwy

We’re having some temporary issues with some of our webmail machines. We believe it has to do with a recent plugin upgrade. Our admin team is currently investigating this issue and hope to have it back to normal shortly. We’ll update this post when we believe the problem is resolved. We’re sorry for any inconvenience this causes you.

Update: 12:15pm – This should be resolved now. The problem wasn’t with a plugin upgrade, but with our Squirrelmail database. Our admins fixed this up a short while ago and we’ve tested several customer accounts to insure that it is working again. If you continue to have problems with webmail, please contact support.


 
© 1996-2008, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
12:08 pm

A spacey DreamHostPS host getting new hardware

Posted (July 4th, 2008 at 12:08 pm PST) by kitchen

One of the DreamHostPS host machines has experienced a hardware failure and we’re giving it some new hardware. The failover process takes a little bit and we’ll have everything back up and running shortly.

Affected DreamHostPS servers:
chunky doplr dreamserver etxwww francis_piragua geeee lareina neptun plainprograms radiosantafe sizzleone tatiana unipar webs86

We will post updates here as things progress!

Update: 12:59pm pacific: The hardware has been replaced and we are currently working on restoring services. Shouldn’t be too much longer before we give the all clear!

Thanks for your patience!
The Happy DreamHost Light Fireworks in the Datacenter Team


1:24 am

A few MySQL servers going down shortly for Maintenance

Posted (July 2nd, 2008 at 1:24 am PST) by Sandon

Around 2 AM PDT (in around 35 minutes) I will be taking down the following MySQL servers for maintenance:

greebo, spike, simba, juno, mufasa, monty, putney

The downtime should only be in the area of 10 minutes but it is possible it could take up to 30 minutes. I apologize for the short notice on this outage. I will update this post when everything should be back up.

Update 2:17 PDT:
Ok, greebo, spike, simba, juno, mufasa, and monty are all back up and should be good. The last one on the list is putney which I will be taking down in about 10 min or so and should be down 5-10 min.

update 2:42 PDT:
Ok, Putney is back up as well now so all services should be restored. Everything should have been back up as of 10 minutes ago but it took me a while to update the status post. If you are still having problems with MySQL service please contact support.


10:55 pm

Putney MySQL is down

Posted (June 30th, 2008 at 10:55 pm PST) by JeffH

Putney mysql went down. Admins are investigating/fixing. We are very sorry for the inconvenience. Please bear with us and check back for updates. We will post them as the situation progresses.

Update: Data is being restored from backup, and machine is being failed over to new hardware. This may take a little while. Will update as soon as it’s resolved.

Update:  After a long night of data restores and hardware replacement, we’re all back up and running now.  If you’re still having problems though, write to support and let them know!


9:07 pm

Randy Cluster Outage

Posted (June 30th, 2008 at 9:07 pm PST) by Jason C

Customers on the randy cluster might notice that they are not seeing their sites currently.  We had a bit of a network hiccup and our admin team is working on cleaning everything up.  Once things are resolved, we will post more information here.

The servers affected by this issue are: alexander arkanoid attila baht bam bob bomberman breakout caesar centipede contra cornelius cortes crack cruisin cssmania deblume defender digdug dinar dinero dollar donkeykong ellipsis euro fidget forint franc gage galaga gap go grant guilder happy herod ikari ike io ivan jezebel joust karnov katamari khan klax korben kroner leeloo lei limbo-randy limbo-randy2 limbo-randy3 lira lucky mark marvin morgan napoleon nexus nimitz orbital pacman paperboy patton peso pizarro pound qbert quid rhod ruble rupee rygar scagnetti schilling scipio shekel slaughter slimy tetris ugh ultima venice vex vito wasabi yen yoda zorg

Update – June 30, 2008, 10:10 PM PST: So our admin team was testing a new network configuration on the randy cluster to improve stability and performance. Unfortunately, the servers did not seem to agree with it at all.  After repairing the network config on all the servers across the cluster, the admin on duty has assured me that things should be back online now.  If you continue to see problems with your server or sites, please contact our technical support department and we’ll gladly work with you on resolving any leftover issues you might be seeing.


3:03 pm

Server ‘tetris’ being moved to new hardware

Posted (June 29th, 2008 at 3:03 pm PST) by andrea

Tetris has been having some issues this morning, and none of the reboots we tried have helped. We’re moving it to new hardware. It shouldn’t take longer than 30-60 mins to complete. Please check back here for updates. Sorry about all inconvenience this has caused.

Update: 4pm PST – server is back online. If you’re still experiencing problems, please contact support with the specifics.


8:07 am

Monty having troubles

Posted (June 29th, 2008 at 8:07 am PST) by DreamHost

The MySQL server, Monty, is currently unresponsive. The datacenter has been contacted and Monty should be back up and running shortly. I apologize for the inconvenience!

UPDATE:  Monty’s back up.  Please contact support if you still have any troubles with your MySQL databases on Monty.


3:58 am

Janky having problems

Posted (June 29th, 2008 at 3:58 am PST) by Brian@dreamhost

There seems to be an issue that’s effecting the janky cluster at the moment. We’re still investigating what is happening, and will update this post with more info shortly. We apologize for the problems!

Update: 04:35 AM PDT

Looks like a file server was having problems, and caused home directories that were mapped to specific locations, to disappear. This has since been fixed, however, the apache web servers that server domains need to be reconfigured. This takes a little time, so domains effected by this should be coming back online in groups. We’re scheduling these reconfigures as fast as our system will let us, so we aim to have everything fixed soon!

Updte: 05:11AMP PDT
Problems with the janky cluster have been resolved. All sites should be up and running again, if you are still experiencing troubles loading your site please contact technical support .


8:28 pm

Krabappel having trouble!

Posted (June 27th, 2008 at 8:28 pm PST) by Brian@dreamhost

It looks like the server “krabappel” is currently having some hardware problems, so we’re going to swap it over to new hardware.

This should take about 45 minutes to an hour, we apologize for the delay, as well as the problems it causes!


7:01 am

Disabled mod_passenger on a number of domains

Posted (June 27th, 2008 at 7:01 am PST) by kitchen

While dealing with some issues with mod_passenger over the last few days, we’ve observed that quite a number of people are enabling mod_passenger on their domains and not actually using it for anything. Because of this we’ve made a restriction for mod_passenger where the path to the web directory must actually have /public at the end, to (hopefully) prevent people from enabling it without any need to do so.

To take care of the existing domains which have been set up in this fashion, we disabled mod_passenger on any currently hosted domain which wasn’t pointing at a /public directory. There are a few of you who might have a symlink to a rails application’s /public directory, you simply need to re-enable mod_passenger through the web panel to get it working again.

If you have any questions, please feel free to contact support, and we can assist you!


7:35 pm

Spacey having some problems.

Posted (June 25th, 2008 at 7:35 pm PST) by DreamHost

There is a complication with the spacey cluster networking. Our admins are looking at the issue right now and we hope to have it resolved shortly.

–Update 8:41pm–
This has been cleared up and all affected servers should be up within 10 minutes. There was a misconfiguration.

Customers on the following machines may have been affected:
7up acquire adhoc adverstore aef agreege albasoun ali alignak alpine amp anadolu andrewslaterpresents ansible appa appletini aquafina aremes arif arizona artica artofmanliness asle ass astore av avocado ayto babblemind ballistic barqs basketmen bawls beebiz benshead bigbrain bigserver blogdu boba bpsites brainsquirt brisk brooklynheights burning cactuscooler cafecompao calpico calvin cappuccino caprisun cc cedar cehennem central cfinc chai charlotterealestate chiller1 chimneyrock choclatique chompy chunky cihan clamato codered coffee coke copping cre8tivegroup crush crystallight culligan cupula curzicorp daddy dads danjacobs darkmirage dasani davros dbe decoratework delphae derby dietrite digitmidia divefilmhd django dominator dominion doplr doubleosting downtown dreamer dreamserver drpepper dshah dymweb dynweb eaglemtnone ecodoc eggnog elroy empire erremedia esc espresso etxwww evian eyedock ezekiel fabsilv fanboy fanta ffxiii fieber fiji filter fishbulb flawless flibble foobar fpccprod frances francis_piragua frappe freerick freewebsites fresca fuze fuzzyraygun fwc g3 galford gatas gatorade geeee geyser gingerale glub gluttony glyn0x001 gnarly goodcow gordini grafia granangular grapico grassroots gravatt grosbouddha gva gws hansens happy2008 haslogics hi-c hiball hithlum hoops horchata hpalumni hstorm hutchtest ibc icq idmo igordon ims inko intenselighting iosrod isabellaval isnmweb izze izzy jackliu jimobi jolt judi keeeeez keiwc kennedydesign kestrel kitchensink kokoserver koolaid koolu krillsc kvailas l0ki ladyanne lareina leanmean lensman lerftacular levkov lift lingolook lipton longcat lorddjinn loronix love lusin mab macnimble madrid magictv mamba maozveg marmol martinellis mastequila maxdps mccreery mega mella mendonca merca mercury_xtreme milk million mimpi minutemaid mirko mmgllc modsquad monster mountaindew mrpibb mtonnies mug musclemilk mystic n00bz navi nb nehi neonblue neptun nesbitt nestea netant netsolution nfwdesign nhym niagara nolaflash nuh odr odwalla oj omsairam onovia opiniuj oran orangebang orthodoxtheologicalschool orvad ovaltine pacifico painame panda pandion pdxpeace pellegrino pepsi perrier perrin pig pinoysoft pixy piyonet plainprograms planetapera platext playitserver plentymag pocapower pointonline pollos polnetwork pom pootify powerade ppps propel protium ps1 pssunkist punch qikserv001 quagmire quixotic radiosantafe rakofi rameniac raoul rawkins rdp reaper redbull redeemer refresco reserver rhetorical rightstuff riptide rocknroll rockstar rocky rojax rootbeer rufio rural sai samhall scarface scary schweppes seagull seawolf seltzer sergnet serpa servebooz seventeendots sharktopus sharpie shasta shindig sigholiday silk silverwolf simpledollar siriusb sizzleone skeenshare skif skywalker slice slim smoothie snapple snidely sobe solarvistas solus sonic sonsuztasarim sparkletts speed4you sprawsm sprite squeebee squirt stacyfayehan stewarts store stormager storyhole sunkist sunnyd superblog supergate superhouse superman tab tajvps tampico tang tatiana tea technomatt tejava thedranskter the_machine threesixty tianjin tino tizer tonic tony trashmenagerie trickbeard tripter tropicana troubled_games trs trux uiopluss unadvised uncletapa underscorewebserver unedune unipar unosaficionados urbart v8 vanguardps vejaisso voss webapp webcomplex webcoot webjts webomega webs86 welchs wellsville whm1 widerfunnel willing wishserver wqli78 xp yoohoo yudomagic zakidesign


 
© 1996-2008, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
6:42 pm

“sarabi:fabia” Experiencing Problems

Posted (June 25th, 2008 at 6:42 pm PST) by Jason C

Some customers on the MySQL server “sarabi” — mostly the ones whose databases are on the service fabia — may be having issues getting thru to their databases.  A member of our admin team is currently working on fixing this matter and we should be able to give you a resolution shortly.  Expect an update with more information here soon.

Update 10:53PM Pacific: Just got word from our MySQL guys that the recovery of this is finished and things should be running normally again. Please contact support if you are still having issues!


6:43 am

A Controller Was On The Fritz

Posted (June 25th, 2008 at 6:43 am PST) by patrickm

We found that one of our controllers has been pushing out bad Apache configurations and we’ve removed it from the rotation until we can get the issue resolved.

If you notice that your domain is not resolving correctly and pointing to a completely different domain, please contact support and we’ll fix that right up for you.

We apologize for the inconvenience this may have caused!


8:40 am

Planned network outage July 11

Posted (June 18th, 2008 at 8:40 am PST) by DreamHost

On July 11, 2008 starting at 10pm we will have a brief network outage to restructure our routing tables. We are planning two hours to deal with any quirks that may come up, however our Cisco network engineer estimates the total downtime to be under 30 minutes.

During this downtime DreamHost will be completely inaccessible, however no data will be lost. Email will be deferred on remote servers and delivered once the network comes back online.

This restructuring is to put the final fixes on various packet loss issues we have applied “band-aid” fixes to in the past, including installation of an extra core router at one of our facilities. Cisco TAC believes they have isolated the problem to duplication of routing information that will require changes to router interfaces to repair. Unfortunately this also requires that those interfaces be reconfigured from scratch, so there will be some downtime while this is done.

Sorry about no updates here, we were updating our “reminder” post. Here are the reminders, just copied and pasted:

Update 7/12/08 12:32am PDT:
The network was back up briefly before being taken down by an unexpected hardware fault on one of the central routers. It’s currently coming back up and with any luck, things will be returning to normal shortly.

Update 7/12/08 2:58am PDT:
Everything’s been back up for a while now and we’ve been monitoring everything very closely. From what we can tell, the previous issues are all gone and everything’s running smoothly. We’ll continue to monitor the network very closely over the next 24 hours to be absolutely certain no strangeness pops up.


8:15 pm

Puncher Got Knocked Out

Posted (June 16th, 2008 at 8:15 pm PST) by Jason C

The puncher file server for users on the postal cluster required a quick reboot this evening.  The cause of this reboot was battery issues and caused services across several servers to become unresponsive for a short amount of time.  Services are being restarted now tho and should be back up momentarily.

Please note that there will be a brief period of downtime for some postal customers tomorrow while we replace the battery in puncher.


12:54 pm

Some servers getting quick reboots today

Posted (June 11th, 2008 at 12:54 pm PST) by jeremy

For security reasons, we’ll be upgrading kernels on some of our servers today, requiring a quick reboot. It’s only about 15% of the servers, so most of you won’t notice any impact. We’ll also be putting some measures in place to prevent unnecessarily long downtimes so total downtime should be in the range of 2-3 minutes maximum for those that are affected. For obvious security reasons, we can’t post a list of servers affected, but rest assured that we’ll be keeping a close eye on them all to make sure no undue downtime is incurred.


10:41 am

Janky Cluster Slowness

Posted (June 11th, 2008 at 10:41 am PST) by Justin@DH

We are experiencing some file server issues with the “janky” cluster which are causing some higher loads and email slowness. Our network file server admins are investigating the cause to get this fixed as soon as possible. Our apologies for the inconvenience. When we have further information we will update this post.

— Update —

At this time the file server is looking better, plus an additional file server is installed that will get some data moved to it as soon as possible to help balance the disk operations and prevent slowness. This will be an ongoing project as it does take time to move data but will help much for stability in the long run. We are continuing to monitor the situation and should we have further info we will post it here.


9:07 pm

Spunky Mail Downtime

Posted (June 10th, 2008 at 9:07 pm PST) by brians

We are currently experiencing an outage in the spunky cluster. This is affecting your ability to connect to the incoming / outgoing mailserver. Our Admins are currently working to resolve the issue. There aren’t any further details at this time. Again, we sincerely apologize for any inconvenience this may have caused. Please check back here for future updates.

UPDATE 9:53 PM PDT: Everything has been resolved. The root cause was the result of an IP change on a few of the machines that handle mail delivery for your domain. Sorry for the inconvenience this may have caused and we truly appreciate your patience.


10:12 am

Upgrading file server gloat

Posted (June 10th, 2008 at 10:12 am PST) by DreamHost

We are upgrading the file server gloat to the next update release of Solaris. This should resolve the crashing issues we have been experiencing over the past week or two. We have tested this upgrade on multiple other machines and it has cleared up the SATA driver bug with which the servers seem to ship.

This upgrade will unfortunately take a few hours to complete, during which time your data will not be available.

Update: 7:00pm Pacific This took longer than expected due to a missing step in the instructions Sun provided. We brought the server up for the latter half of the day, however brought it back down again once we had the missing step confirmed. Everything should be smooth sailing now. Please contact support if your website is still down. I apologize for the extended downtime.


3:07 am

Janky being, well, Janky

Posted (June 9th, 2008 at 3:07 am PST) by Terri

One of the file servers is having issues serving NFS requests to servers in the Janky cluster. This could potentially impact users on the following clusters:

adonis agamemnon ajax aphrodite apollo aries atlas baachus barefoottours calypso charon charybdis chronos coolblue daedalus dione dionysus electra erebus erida eros ganesha gorgon hades hecate hector helios hephaestus hera heracles hermes hestia hypnos icarus iris jove kratos leto medusa midas minerva morpheus nemesis odysseus oedipus osiris pan pandora pegasus persephone poseidon proteus rhea sisyphus tartarus tethys thanatos thetis triton tyche zeus

This will effect mail and web sites, causing downtime. Our admins are working as fast as possible to diagnose this problem and we will post updates as soon as we know what is causing the issue.

update It looks like the file server did not want to deal with some number crunching. It has been adjusted and everything is working fine now. If you have any further issues, please contact tech support. Thank you for your patience!


10:31 pm

Blingy’s Lost Its Luster

Posted (June 8th, 2008 at 10:31 pm PST) by Jason C

Folks on blingy might be experiencing some issues with their sites coming up right now.  The main network file server on the cluster was in need of a reboot and was causing some problems for customers.  Once the issue is resolved we will update this entry with as much information as we possibly can.

Update: A file server other than the one that our admin team diagnosed as problematic was preventing mounts from completing. That file server has recovered and is plugging along again. If you continue to have issues, please mail technical support so they can assist you with resolving them.


 
© 1996-2008, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
7:36 pm

2 janky file servers rebooted

Posted (June 7th, 2008 at 7:36 pm PST) by kitchen

Two of the file servers serving the janky cluster have rebooted themselves due to a hung process working on the filesystem. They are both back up and serving files again, and we are working on cleaning things up. We will post an update when we feel everything is back to normal.

Update: one of the two filers had not saved its running configuration and was missing an IP address so it was not actually serving files to servers in the cluster. This caused mounts to fail and apaches to fail. This has been resolved and things are coming up. We’ll post the all clear shortly!

Update: everything appears to be back to normal. If you are still having issues please contact support!


4:53 pm

Filer Trailer Down

Posted (June 7th, 2008 at 4:53 pm PST) by Leon

The “trailer” filer is currently down and we’re attempting to bring it back up! We will update everyone as soon as we’ve brought it back to life! If you are on any of the following servers, you will be affected by this:

afree annie apu arnie bart bigredgiant bitters brandine busycards carl cheeky cletus dansweb dawber dib drederick eclipse flanders gir glass gloop harpo hibbert homer jimbo kearny krusty lisa lovejoy marge mcclure misfit mona ninja osteele otto quimby quimbys raiden ralphie rwn salt sanjay seamus skinner taipan tak tooth wiggum willie yerba zim

Update 5:19 PM PDT: The filer is back up and we’re doing a bit of clean up. We will update everyone once more when everything is good to go!

Update 5:42 PM PDT: We’ve brought everything back up on the filer and all appears to be normal again. If you’re still experiencing issues please contact support!


10:53 am

Email Hiccups

Posted (June 5th, 2008 at 10:53 am PST) by Jason C

We experienced some brief problems (no more than 20 minutes) with email accounts being inaccessible due to a permissions error. The issue was fixed fairly quickly once it was noticed tho and we apologize for any issues this might have caused.

If you feel you have lingering issues because of this matter, technical support will be happy to help you out in whatever way that they can.

Note: Some email might have been rejected due to the directories not having correct write permissions — so if you were expecting something important during this period, it would probably be best to have the message sent again.


1:34 am

Spree died

Posted (June 5th, 2008 at 1:34 am PST) by Sandon

The web-server “spree” has died. It appears that it blew a power-supply but it was a very old machine and the cause could be something else (It simply won’t turn on). I went ahead and started failing over “spree” to a standby server with much newer hardware which should take 30-45 minutes. In the meantime all sites hosted on “Spree” will be down until the fail-over completes. Once completed everything should be back up and all the sites on “Spree” will be on upgraded hardware.

Update 02:10 PDT: 90% of the services hosted by this server should be back on line and the last few are coming back up now

Update 02:18 PDT: Ok, this server should be completely back up an all service restored. If you are still having problems with your site and your site is hosted on ‘spree’ please contact support.


3:05 pm

Network Problems

Posted (June 4th, 2008 at 3:05 pm PST) by brians

We are experiencing a major network outage at the moment. This will affect your ability to access your site, email, etc. Our admin team is fully aware and working to resolve the issue as quickly as possible. Unfortunately, we do not have more details while the admins get to the bottom of this. We sincerely apologize for any inconvenience this may have caused. Please check back here for updates as this issue progresses.

Update – June 4, 2008 @ 3:22 PM PST: It would appear that someone in one of our data centers wrongfully turned off one of our circuit breakers.  This caused a bunch of central machines to shut down as well as one of our network distribution switches.  Once this was noticed, our admin team sprung into action to take care of this.  Our servers (and most of your sites) should be coming back online now and we’ll update this post when things have leveled back out. -jasonc.

Update – June 4, 2008 @ 3:50 PM PST: One last update before the “all clear”.  The major delay in getting all our services back up (web panel, webmail, etc.) is due to the fact that our central database is recovering from the outage.  Once it does so tho, we should be able to safely mark this issue as “resolved”. -jasonc.

Update – June 4, 2008 @ 5:34 PM PST: And we’re back!  The database should be plugging along and the web control panel, webmail, WebFTP and all Easy One-Click Installs should be running again. If you continue to have issues, please contact Technical Support and we’ll be glad to sort things out for you. -jasonc.


12:12 pm

SSH being upgraded everywhere

Posted (June 4th, 2008 at 12:12 pm PST) by jeremy

To better protect everybody from the recent openssl holes that Debian introduced, we’re upgrading our SSH packages to a newer version that allows us to blacklist public keys that were generated using the broken systems. This new blacklisting only affects people using public key authentication (ie if you use a password to log in, this won’t affect you). If you _do_ use public key authentication and suddenly are unable to log in without a password, it’s probably because you generated your ssh key on a broken Debian or Ubuntu box and will need to generate a new ssh key.


2:03 pm

Major network problems (Resolved)

Posted (June 3rd, 2008 at 2:03 pm PST) by danwy

We’re currently experiencing some major networking problems. Details are sparse at the moment, but our admin team is currently investigating it and we will be updating this post as soon as we have any additional information. We’re sorry for any inconvenience this causes you.

Resolved 2:15PM Pacific: It seems that bringing online a new switch messed up our VLAN definitions on the core router. This meant routing was knocked offline until the VLANs were restored. We have rectified this, and setup the appropriate precautions so that a non-core switches and routers cannot update our vlan tables.


7:22 am

Randy cluster having problems. (Updated)

Posted (June 2nd, 2008 at 7:22 am PST) by David

Unfortunately we’re experiencing some sort of problem on the “Randy” cluster which is effecting web and mail services. Many customers have reported slowness with their web sites. Our admins are looking into this now and will update this post when we have more details. We sincerely apologize for any inconvenience this may be causing.

NOTE: You can determine which cluster you’re on via our control panel. When you log in with your WebID click on the “Account Status” button near the top-right. The cluster your account is on is listed as “Your Email Server”.

The “Not-so-happy-Monday-morning-fix-it-team”.

UPDATE: 8:40 PDT – Looks like there was a problem with one of our file servers (Janice). It seems that the problem has been resolved and we’re restarting the effected servers. Things should be back to normal again very soon.

UPDATE: 11:15 PDT – The admins tell me that the problem has been resolved and that all but the most stubborn servers are back to normal again. If you continue to experience any problems with web, mail or database and are on the Randy cluster please submit support requests via the normal channels.


3:19 pm

Lipton moving to new hardware

Posted (May 30th, 2008 at 3:19 pm PST) by Leon

The lipton web server is currently experiencing some issues, so we’re moving it to new hardware. All services will be back up shortly!

Update: It’s been back for quite a while now. Sorry for the delayed update to this post!


7:23 pm

Skimbleshanks MySQL Unresponsive

Posted (May 25th, 2008 at 7:23 pm PST) by glen

MySQL server Skimbleshanks was found to be unresponsive. Attempts at remote and local reboot have not brought it back up. An admin is being sent onsite to get it up in as expedient a matter as possible.

Update
It’s back up


 
© 1996-2008, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
3:03 am

Spunky PS host machine disk failure

Posted (May 23rd, 2008 at 3:03 am PST) by Sandon

This isn’t the same machine as a couple of days ago. If your PS is named one of the following:

dilithium geekwave h2o herock mitchbox otnet plastica redaau sailormoon sephy starreserver tatooine uptownelite wiredgeek wozie

Then your sites will be down for the next hour or so while we get everything transfered to new hardware. If you have downtime and you are not on one of the following PS servers then this outage does not effect you.

Update 4:58 PDT:
Ok, everything has been moved to the new server and all sites on the PS’s above should be back up.


2:12 pm

Power Outage – Generators working

Posted (May 22nd, 2008 at 2:12 pm PST) by Ralph

One of our main data centers is unfortunately dealing with a power outage at this time.

We are in the clear at the moment as all of our servers are still up. We will update you as soon as the building has this issue resolved. Nothing will go down because of this issue, if there is downtime caused by this we will update this post.

Update: We are back on city power. If you have downtime issues it was not related to the power outage. This particular incident did not cause any service problems.


5:58 pm

Filtered Mail Delays

Posted (May 21st, 2008 at 5:58 pm PST) by Jason C

Customers with the Junk Filter enabled for their domains have been reporting delays due to an issue with our filtering machines. The team here has actually ferreted out the problem already and the number of queued messages is dropping as they process thru the filters. There will still be some delays with messages while the mail gets back under control — but please know that we’re doing our best to sort this matter out as quickly as possible.

Once things level back out, we’ll update this ticket with an “all clear”. Sorry for any issues this might have caused!

Update Fri May 23 11:48:44 PDT 2008 : We are continuing to monitor this situation and things are improving. Some delays are still happening but not as frequently or as severely. We know why the delays are occurring but not the exact root cause. We are now in the process of upgrading all of the core software in use on our mail filtering servers to the latest versions.

Update Fri May 23 17:04:07 PDT 2008 : The queues have leveled out to a manageable size and are processing mail normally and quickly.  If delays persist at all, please let us know by writing into support and we’ll gladly take care of you as best we can.


2:30 pm

Blingy problems

Posted (May 21st, 2008 at 2:30 pm PST) by Ralph

Some of you are well aware of the problems we have been dealing with on the blingy cluster:

http://www.dreamhoststatus.com/index.php?s=blingy

The troublesome file server is causing problems again. The majority of issues today were caused by our efforts to completely offload the problematic file server onto more reliable hardware. Our moves, unfortunately, overloaded the file server. We have cut back on the amount of concurrent moves we are running and blingy servers are catching back up again. Unfortunately, this issue will not be resolved until we get the problem file server completely out of the loop, which is a slow process.

Update: Just wanted to note that if you are in the blingy cluster, submitted a support ticket but have not received a reply do not fear we are working on your problem. We are moving everyone who has written in so once we have an update we will reply to your message. Sorry for the delay in the support tickets.

Update 5-22: Some of you have been writing in that your data is missing from the wycliffe glob. This was due to a move that did not complete correctly. The good news is that we still have all of your files and everything will be appearing again shortly. The bad news is that you will still be on the old file server. Once your sites are working again, we will be starting the migration over (which should run seamlessly in the background) to get you off of titanic and onto a stable netapp.

Update 5-26: We were able to complete a decent amount of moves this weekend but the process is still ongoing. For those of you on wycliffe there is still some moving to be done. We thought wycliffe would be done today at some point, but it seems like a few more days of moving before that is complete. We thank you for your patience as we move you to the new file server.

Update 5-27: Problems have increased today unfortunately as our moves keep running. The problem is that after we move the data, we have a system that goes through and makes sure there was no lost data. So moving data essentially takes 3 times longer than your normal copy. With the wycliffe glob, the biggest glob in Blingy, this turned out to be a major issue as we are not actually removing the data from the server until the data is all moved over successfully. This means that even though data is moving, it is not freeing up the space we need for the problem file server to work correctly. So right now we are working on the accounts that have already moved, and pause the rest of the move so that we can move some smaller globs faster. So we are still running moves, we are just not removing data of the problem file server fast enough which is unfortunately causing the problems currently being experienced in the blingy cluster. We will get his calmed down as fast as we can and continue our efforts.

Update 6-12: For the most part blingy problems have been limited to minor issues. We are still continuing the moves that will get the problem file server out of active service. This is a lengthy process, but so far moves have been successful and service is getting better.

Update 7-30: The blingy cluster has been stable for over a month. Although we are continuing to offload the blue arc (it’s HUGE), there are no current issues with the job it’s doing. Please report any issues you have with your service to support and thank you very much for bearing with us through this.


3:05 am

a few spunky DreamHostPS servers moving to new homes

Posted (May 21st, 2008 at 3:05 am PST) by kitchen

The hard drive failed in one of our DreamHostPS host machines and we’re working on moving it over to new hardware to get things back up and running smoothly.

The affected servers are:
bombadil boybutter burn diaryoapg emn headhunter muhammadiya poto sugardoodle tommy_the_cat

If you are on one of those machines, please be patient while we work to get them back up and running.

We will post updates here as things progress.

Update 6:12am pacifit: Sorry for the additional delay on this matter. The target machine we were moving these all to also had hardware problems, and we had to unbox and prep a new machine for them to get put on to. We will continue to post updates as things progress.

Update: 7:57am pacific: Things are coming together now, we should have everything up on these very shortly.


5:38 pm

Recent SSL vulnerabilities

Posted (May 20th, 2008 at 5:38 pm PST) by jeremy

It seems that a lot of you have been worried about the recent SSL issue that Debian has had based on emails that GeoTrust has been sending out. Thankfully, our internal servers that handle the SSL key generation had not been upgraded to the vulnerable Debian release because we hadn’t felt it necessary to introduce the possible new problems that go along with an OS upgrade. That means that all of the SSL (and ssh) keys that we have been generating for the past couple of years are unaffected by this bug and still completely secure.


8:46 pm

spunky-vserver21 moving to new hardware.

Posted (May 16th, 2008 at 8:46 pm PST) by DreamHost

This is currently being done and should be resolved shortly.

Update — It took longer than anticipated, but is working now.


1:38 pm

Dietrite moving to new hardware

Posted (May 16th, 2008 at 1:38 pm PST) by Leon

The dietrite web server is having issues on boot, so we’re moving it to new hardware! All services will be back up shortly.

Update: Dietrite is now back up on new hardware and running happily! If you’re still experiencing issues, please contact support.


1:37 pm

Rock being moved to new hardware

Posted (May 14th, 2008 at 1:37 pm PST) by DreamHost

Webserver rock is having trouble booting, so we’re going to be moving it to new hardware. This process should take about thirty minutes.

Update: this has been finished and all sites hosted on rock should be up and running, if you have one that’s not please contact support.


12:02 pm

New Feature: Passenger (mod_rails)

Posted (May 13th, 2008 at 12:02 pm PST) by Dallas

As of today, all DreamHost customers can enable Passenger (mod_rails) for Ruby on Rails applications.

Briefly, all you do is enable the Ruby on Rails Passenger (mod_rails) option for any existing or new web domain in the DreamHost web control panel. When you then point that domain’s web directory to the public directory of an existing Ruby on Rails application it will work automatically. For more detailed information, check out our Passenger wiki page. You may also want to read additional details in the DreamHost blog post.


 
© 1996-2008, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
11:24 am

Brief outage for mysqls stimpy mimsey toonces roary and snowball

Posted (May 13th, 2008 at 11:24 am PST) by DreamHost

We need to do some maintenance on the mysql servers stimpy mimsey toonces roary and snowball, so this will be unavailable briefly.

We’ve finished this maintenance and all of these servers are up.


6:14 am

Clifford blew a Drive!

Posted (May 13th, 2008 at 6:14 am PST) by Sandon

The MySQL server ‘clifford’ has blown a hard-drive and I had to replace it and the array is rebuilding as we speak. This should take approximately 30 minutes but the server will be down/inaccessible in the meantime. This should only effect people with ‘clifford’ as their MySQL server. You can find this out from Goodies -> Manage MySQL as your server will be listed in the format of ‘servername:service’ and if your ‘servername’ is ‘clifford’ than this is effecting you.

From the Happy DreamHost NightCrew!

Update: 8:18 PDT.

Clifford is having problems with building the array so one of our senior admins has been contacted and is working on resolving the issue.

Update: 11:25 PDT.

We are still working on fixing clifford and apologize for the continued outage. There’s currently no eta on when it will be up, however our admin team is hard at work to get it going as soon as possible.

Update: 12:11 PDT.

We’re now in the process of moving clifford to new hardware and restoring from backups. Databases hosted on clifford should start coming online soon, though the entire process may take a few hours.


11:13 pm

Bitters down for the count!

Posted (May 11th, 2008 at 11:13 pm PST) by Brian@dreamhost

Bitters seems to have lost it’s local drive, so we’re moving it to new hardware. Domains on it should be working within 30-45 minutes or so.

We apologize for the problems this causes!

Everything should be back up and running. If you’re still having problems, please let support know!


11:56 am

Blingy Slowness Issues

Posted (May 10th, 2008 at 11:56 am PST) by mir

We’ve continued to offload the problematic file server and loads across the cluster have actually looked quite good (almost all servers under a load of 5, many around 2-3 and only one was ever at or above 10) so we had hoped that we could restore snapshot backups for users on that server. Unfortunately the result increased the disk usage again on that file server to the point that is has been displaying the same symptoms as before. The good news is that the fix is simply a matter of disabling the backup snapshots again and then dropping that data. We’ve already done the disabling and are in the process of dropping – this is resulting in some very inflated loads across the cluster but as soon as we have completed this I’ll be issuing soft reboots (easier on the hardware) that will fix the loads you’re seeing as well as adding an update here. This should eliminate the performance issues that have been reported (we’ll just have to get that particular file server even lower before we restore those snapshots again). My apologies for this and for the concerns that it has caused.

Update: The utilization is already down from 93% to 92% but this is multiple TB of data at issue so it may take a few hours to complete the dropping of the data. I’ll be keeping an eye on it and rebooting any servers that appear to need extra attention (changes like this often result in that extra step being needed to stabilize things). There will be another update once we’re satisfied that things are fixed.

Update: We’re now back down to 89% – I am still seeing load issues so we’re not out of the woods yet but once the process completes we’ll know if that did it or not (I will continue to update).

Update: Utilization is at 87% and loads have improved so I am updating the severity to medium but not marking it resolved as we still need to see the loads and performance back to what they were a week ago.

Update Tuesday, May 13th: Utilization crept back up (it appears that there were still some rules on the file server leading to creation of the snapshots we deleted, these have been removed and the deletion process is underway again). The admin team is also looking into completely removing this file server from the system (for now we intend to keep offloading it non-stop to ensure that no more issues crop up).

Update Thursday, May 15th: The snapshot data is gone and loads have dropped back to what they should be but we’re going to be moving people off of that file server (we have new hardware coming in that should allow us to completely offload it and eventually scrap it).

Update May 21st:  Looks as though the offloading of the filers got a bit backed up over the weekend and has caused the utilization to go back up to the point where it is causing issues again. We are continually doing offloads and working on the servers that are being affected.


8:06 am

Failing over Milk

Posted (May 10th, 2008 at 8:06 am PST) by Sandon

The web-server ‘Milk’ is being failed-over due to bad hardware causing it to crash a couple minutes after it boots. It is being moved over to new hardware and the move should be completed within 30 minutes. This should only effect people with ‘milk’ as their webserver. You can find out if milk is your server by clicking “account status” in the panel and it would be listed as “Your web server”

Edit: Turns out this was a bad site on the server causing it to crash in strange ways. This site has been disabled and your sites should be back online!


8:27 am

Emergency OS patch on file server

Posted (May 9th, 2008 at 8:27 am PST) by DreamHost

Per an open ticket with Sun we need to apply two patches to one of our file servers. This is to hopefully fix a degraded zpool which will not finish a parity rebuild. There are exactly 78 users in the frisky cluster on this file server. This will bring your email and web services offline for the time being. The patch should only require about 15 minutes of downtime. I apologize for doing this patch during peak hours, but we really need to get this data back up to full integrity.

Tech nerd details: The raid array is a raidz2 operating with one failed disk. It has been rebuilding off of a hot spare for a day or two and reset the rebuilding process itself after getting to 99%. We contacted Sun and after analyzing troubleshooting information believe a kernel + ZFS patch should resolve the problem. Fortunately this is a raidz2 so it can sustain a disk failure and still be fault tolerant.

Update: Well, one of the two patches was installed. It seems SunSolve is rejecting our service contract to download a specific patch in the dependency tree. We’ve updated our case with Sun. The system is back online and serving files!


7:14 pm

DingDong and Pizarro down

Posted (May 8th, 2008 at 7:14 pm PST) by glen

The HTTP servers DingDong and Pizarro are both currently unresponsive to our reboot efforts. We are working on getting manual reboots done in their respective data centers or evaluating whither moving to new hardware is necessary. Estimated downtime as long as 1 hour.

Update 7:44p
Pizarro is back up and seemingly stable on new hardware
DingDong is awaiting a tech to reach it’s data center still.

Update 8:15p

DingDong has been manually rebooted and is up. Tests of sites show them responding. If you have any further issues related to either of these 2 servers, please contact support through our panel.


4:45 pm

Central database crash

Posted (May 7th, 2008 at 4:45 pm PST) by DreamHost

Our central database server crashed and restarted itself. It is currently replaying transaction logs and should be back in under an hour. This should not affect your websites, email, etc, but the user control panel (https://panel.dreamhost.com) and similar services are down until it comes online.

We are monitoring the situation and will report back where when it comes online!

Edit: And webmail! I forgot those two were tied together. Regular IMAP/POP3 email access should continue to work.

Edit: 5:28PM Pacific It looks like we’re back in business! The user control panel and webmail are working. We will continue to check the rest of our central services and update this if we find anything else still broken! If you are having problems, please contact technical support.


9:49 am

FTP problems (connection drops)

Posted (May 7th, 2008 at 9:49 am PST) by andrea

Some of our customers are experiencing problems related to their FTP service. This includes error messages while connecting or dropped connections. We’re looking into it, and will post an update as soon as we know more. Sorry about the inconvenience.

Please check back here for updates.

Update 5/7/08 11am Pacific — We have narrowed down the problem to only affecting customers using the following ISPs: AT&T/Sbcglobal, Comcast and RoadRunner. We have tested the following solution with the help of one of the affected customers, so this should work for all: You will need to change your ftp-only user to sftp to go around what we suspect to be bandwidth throttling affecting only FTP connections.

Edit your ftp-only user to be sftp in the control panel under Users/Manage Users ( https://panel.dreamhost.com/index.cgi?tree=users.users& ). This should schedule a user update, and as soon as the little clock next to your username in the control panel is gone, log in with that user using the SFTP settings on your upload client, and you should be able to upload the large files. If your current client doesn’t have SFTP connection, you will need to download another one.
Please note, if you’re already using a shell user to connect through ftp, then all you need to do is change to sftp in your client, you don’t need to modify your user.

If the above doesn’t make your problems go away, then it’s most likely unrelated. Please send support a message, so we can look into it for you.

Update 5/8/08 9am Pacific — Apparently Verizon and other ISPs have been added to the list. Even if your ISP is not listed, please try to change to sftp, and if that doesn’t work, let us know with as many details as possible.

Update 5/8/08 10:50am Pacific — We’re continuing to look into this problem to make sure it’s not something on our end that’s causing the connection drops. We have just tested a few more cases were the customer switching to SFTP didn’t make a difference, but so far we couldn’t replicate the problem, but we’re still checking, examining more data.
In the meantime, continue trying to switch to SFTP to try that way. We recommend that over FTP anyway for its security. If you’re using telnet, you should be switching to ssh instead.

Here’s a link to the article in our Wiki regarding the use of SFTP…
(http://wiki.dreamhost.com/SFTP)

Update 5/09/08 8:15pm Pacific — We’re still looking into this issue, and thought it was resolved for most users, however, that doesn’t seem to be the case. At this point, all tests we’ve run on users mentioning problems come back working 100%. We’re unable to verify the problem at all, other than the support that’s coming in about it. We do have some support personal on some of the effected networks, and will be doing tests.

We apologize for the delay in getting this fixed, however, it’s not reproducible on our end, at all, so that makes it even that much more difficult to track down. We’ll keep updating, the more info we have.

Update 5/10/08 7:45pm Pacific — Ok, this still seems to be happening, however, we may have found a way around things. We’ve tested on a known problem network, and the test went through correctly. Please try FTP again, as some head way in the issue has been made. If not, hold tight, we’re still working on the problem

Update 5/15/08 11:00am Pacific — We’ve made a couple of small changes to our FTP server that seem to be resolving the connection dropping issues for some people. The changes should now be live on all shared servers and will be live on all PS servers shortly. If people could test some of their uploads and report success or failures to support, the help would be appreciated.


9:25 am

Webserver Hermes being moved to new hardware

Posted (May 7th, 2008 at 9:25 am PST) by chih

Hermes crashed earlier and server isn’t coming back up when rebooted. It’s currently being migrated over to new hardware and should be back up shortly.

Sorry for the inconvenience this has caused.

UPDATE: Migration is complete. Your sites should be back up and running. Contact support if your sites are still down.


 
© 1996-2008, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
3:26 pm

Email server emergency maintenance tonight (janky,randy,postal,spunky)

Posted (May 6th, 2008 at 3:26 pm PST) by jordan

About 30 minutes ago we had some major problems with one of our email load balancers that keeps email for janky,randy,spunky and postal clusters. Tonight at approximately 11:30 PM PST we will be doing some maintenance that may affect the performance / uptime of any customers that have email in these clusters. It is only expected to last about 10-15 minutes at most and we apologize for the short notice. This post will be updated as soon as the maintenance has completed.

UPDATE 11:30 PDT
Maintenance has completed all mail services should be up and running again. If you are still having problems please contact support.


11:18 am

Problems caused by apache service updates

Posted (May 6th, 2008 at 11:18 am PST) by andrea

We have noticed that any apache service updates initiated from the control panel are breaking the service. This includes anything that has to do with the domain’s web service, like adding a new domain, changing FTP users on domain, etc. This doesn’t just break the actual domain that initiated the change, but all domains on that apache service.

We are fixing this as we find them, and trying to catch up with the ones that broke a little while ago. Very sorry about the web downtime this has caused you, and please check back here for updates.

Uptdate 5/6/08 5pm Pacific: This issue is now resolved. Sorry about the delay, we have reconfigured all apache services for which the firs fix didn’t take. If you’re still having problems, then it’s probably unrelated. Please write to support with some details, and we’ll be happy to check into it for you.


11:47 pm

Delay in quarantined junk mail delivery

Posted (May 3rd, 2008 at 11:47 pm PST) by andrea

We have noticed that messages that were quarantined by our junk filter are not getting delivered to the Junk Folder. We checked on the messages, and they are still on the servers. The reason they weren’t getting delivered was due to a connection problem to the mysql database that controls the Junk Folder. The problem is now fixed, but it will be a while before it all catches up, as mail has been backed up since last night. We apologize for all inconvenience this has caused. We’ll post updates on the progress of junk mail delivery as soon as we have them.

Update 05/04/2008 noon Pacific — Quarantined junk mail is again being delivered to the Junk Folder. Sorry about the delay.


12:06 pm

redhot getting new hardware

Posted (May 2nd, 2008 at 12:06 pm PST) by kitchen

Redhot has had a hardware failure and needs to be replaced. We are currently working on moving it over to new hardware.

This should take about 30 minutes, and we will keep you posted as things progress!

Update: 1:06pm pacific: the failover has completed and services are up and running. If you experience any issues, please contact support!

Update May 3, 2008: It looks like the new hardware was not set up properly so for now so we’ve moved to another backup server that was available (I am ensuring that this machine is configured correctly for you). The changeover is almost complete.


9:47 pm

Mysql Maintenance

Posted (May 1st, 2008 at 9:47 pm PST) by DreamHost

These mysql servers will be experiencing some downtime tonight for maintenance:
jake
leo
midnight
snarf

Downtime is expected to be less than 30 minutes. It should happen shortly after midnight PDT.

–Update
The maintenance was preformed and the servers were back up within 15 minutes.


7:27 pm

Dr pepper moving to new hardware.

Posted (May 1st, 2008 at 7:27 pm PST) by DreamHost

There’s an issue with the hardware on dr pepper. It is being moved to new hardware now.

Update–
Things should now be working properly.


11:06 am

Loony Mail issues.

Posted (May 1st, 2008 at 11:06 am PST) by mir

I am opening a new post for you guys – we did install an additional file server for your cluster but the hardware proved to be faulty and that’s why you haven’t seen the improvement that we had anticipated. We have our storage specialist at the data center today working on this and we are hoping that he’ll have it up and running after the visit. If that is the case we’ll only need a day or two to offload Clunker and have your mail back to full speed. I will update this post once I have more details.

Update Frida, May 3 2008

The new hardware is up and we are balancing the data so that should be done within the next couple of days and the mail issues you have been seeing will be resolved.


9:42 am

Planned outage for mysql’s toto rover dogbert dino beethoven bootsy camille jaws

Posted (April 30th, 2008 at 9:42 am PST) by DreamHost

On the night of Tuesday May 6th we’ll be replacing one of the network switches that serves the following mysql servers:

toto
rover
dogbert
dino
beethoven
jaws
camille
bootsy
spacey-vsql01
frisky-vsql01

We hope to be able to do this one server at a time with only seconds of downtime for each, however we’re posting this in case that’s not possible and we need to bring all the servers down.

Update: This switch was successfully replaced last night.


4:45 pm

New Charities Tomorrow

Posted (April 29th, 2008 at 4:45 pm PST) by Josh

Just a notice to let you know tomorrow we’ll be changing the charities we have up at our Give to Charity area, so if you want to donate to Kidd’s Kids or the MIND Research Institute and have it matched by us, step up now!

So far we’ve received 21 donations for a total of $491.11 to Kidd’s Kids and 12 donations for a total of $274.67 to MIND.

MORE!


11:17 pm

Mysql server Sanspoof having problems

Posted (April 28th, 2008 at 11:17 pm PST) by jordan

We are currently experiencing problems with the mysql server “sanspoof” crashing. The issue is being looked into at this time, we apologize for any trouble and will update you as soon as there is more information regrading this problem.

UPDATE 23:22:43 PDT

It appears that the memory in this server is causing it to panic and crash. We are going to be replacing the memory and that should fix the problem. This update should only take about 20 minutes.

UPDATE 12:54:29 PDT

Problem has been resolved by replacing the bad memory. All mysql service on Sanspoof should have returned to normal. If you are still having problems please contact technical support.


 
© 1996-2008, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
2:49 pm

Spunky cluster mail issues

Posted (April 28th, 2008 at 2:49 pm PST) by andrea

If you’re on the spunky cluster (check the name of your mail cluster under Account Status in the upper right corner of the control panel), you may have been experiencing delays in sending and receiving mail for the past couple of hours. The mail database for the cluster is having issues, and it’s being worked on. Sorry about the inconvenience! Please check back for updates.

Update 3:30pm Pacific time — This is now resolved, and we’re continuing looking into the reason for the frequent backup mail database issues that have caused the problem. If you’re still seeing issues with your mail, it’s likely unrelated, so please let support know!


12:15 pm

Webserver flamenco being moved to new hardware

Posted (April 28th, 2008 at 12:15 pm PST) by andrea

The web server flamenco didn’t come back after a multitude of reboots, so we are migrating it to new hardware. It should be back up within an hour. Sorry about the inconvenience this is causing you, and please check back here for updates.

Update: Some networking issues caused flamenco to be down for longer than expected, but we’ve got it going now. I apologize for the outage.


9:21 am

Blingy File Server Issue

Posted (April 28th, 2008 at 9:21 am PST) by Justin@DH

We are currently experiencing issues with the main file server for the blingy cluster. Our file server admins are on their way to the data center to look into the issue, and we are doing our best to get this fixed up and working again as soon as possible. This issue will cause websites and email slowness or unavailability until it is resolved. When we have further information or it is fixed we will post an update.

— Update Mon Apr 28 10:06:16 PDT 2008 —

The file server is back up and working, we are going around fixing up any web servers which are still having issues or needing a restart after the file server issues. The file server admins are continuing their investigation into the cause of the file server problem.

Update 4/28/08 12pm Pacific — All servers are back up and running (except for flamenco, see post above). Sorry again about the inconvenience this has caused you. If you’re still seeing problems with your web or email, please let support know as it’s likely unrelated.


9:43 pm

Looney Cluster having issues!

Posted (April 26th, 2008 at 9:43 pm PST) by Brian@dreamhost

We’re currently investigating an issue on looney that’s causing web and mail service to become spotty, and more likely, completely down. This effects the following servers:

apok axl blanka brazil charm curly dhalsim egon guile limbo-looney moe slappy soy tank tofu vega winston

We apologize for this, and should have things working soon!

UPDATE: As of 10:15PM PST, this should be fixed now. If your site is still down, please email support and let us know!


3:18 pm

Snocap Downtime

Posted (April 25th, 2008 at 3:18 pm PST) by DreamHost

The snocap server has been plagued with some intermittent issues. We’re working to resolve this as fast as we can, and will hopefully have more information on this shortly. If you’re on the snocap server you could be affected by slow website performance, website showing as down, slow uploads and downloads, and trouble updating your dynamic websites.


1:55 pm

Central Frisky Server Down

Posted (April 25th, 2008 at 1:55 pm PST) by mir

Customers on the Frisky cluster are experiencing a range of issues:

-slow site performance due to high loads
-slow data access times
-server crashes
-slow/unavailable mail

One of the file servers in the Frisky cluster has crashed – the admin team is working on getting it back up but due to the nature of the server in question it is effecting all servers that mount data from it (which includes many of the hosting servers and all of the mail servers in that cluster). We’ll update here once we have it fixed (if we cannot bring it back up within a reasonable amount of time we’ll see about removing the mounts so that it has less impact). While this is a terrible situation it has allowed us to pinpoint the cause of similar performance hiccups that we have been seeing with this cluster and should allow us to resolve those for our customers. We will update this post as soon as we have more information for you.

Note: if you would like to determine if you are on this cluster just go to your panel here:

https://panel.dreamhost.com/index.cgi?tree=domain.manage&

and click on the ‘DNS’ link for any domain hosted there and scroll to the bottom, the mail MX records will have the name like so:

mail MX 0 mx1.balanced.frisky.mail.dreamhost.com.
mail MX 0 mx2.balanced.frisky.mail.dreamhost.com.

UPDATE 16:23

We are currently moving the data drives from one file server to another. We are still a few hours from resolution on this issue. As soon as we have the drives moved over and the new hardware booting up correctly we will update this status post again.

UPDATE 17:26

This issue is now resolved. We have replaced the faulty hardware and moved your data drives over to the new file server. This should be completely resolved for you.


11:30 am

WordPress 2.5.1 is out!

Posted (April 25th, 2008 at 11:30 am PST) by Jason C

A pretty major and important security fix is out for the latest version of WordPress — and it is suggested that you upgrade your sites via our One-Click Installer as soon as possible. For more information on the upgrade, it is suggested you check out the WordPress Development Blog.


10:51 am

Moving redhot onto new hardware

Posted (April 25th, 2008 at 10:51 am PST) by DreamHost

The webserver redhot has been having some stability issues lately, so we’re going to move it to new hardware in hopes of fixing it. This process should take about 30 minutes.

Update: This was finished at about 11:30am.


7:33 am

Looney Mail Issues

Posted (April 25th, 2008 at 7:33 am PST) by mir

I apologize for the late post but I wanted to make sure that we had a resolution for you before posting here. Last week I saw complaints about slowness and timeouts. With some really excellent feedback from customers I was able to pinpoint when the issues were occurring and keep an eye on it and with the help of the admin team we determined that it was due to a file server that needed a bit of offloading. We had the hardware available to add 4 TB of space to the cluster and are installing that today so we just need to remove some data from the troublesome file server and all will be well. This is expected to be finished within the next 7 days so by then mail should be working properly again (it may even be sooner than that depending on the point at which it will return to normal). This should also give the cluster enough space to be fine for the next few years (this is an older cluster which grows slowly as no new customers are being added) so you guys are in great shape for a long time to come!


1:56 pm

Central Database Issues

Posted (April 24th, 2008 at 1:56 pm PST) by mir

Some bad code caused our session database to back up which is resulting in errors and sluggish account control panel performance. The admin team is fixing this up and should have it resolved in the next 10 minutes or so.

Update: this issue is now resolved


 
© 1996-2008, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
3:11 pm

Webserver persephone being moved to new hardware

Posted (April 23rd, 2008 at 3:11 pm PST) by DreamHost

The webserver persephone has experienced a hardware failure and is being moved to new hardware. This process should take about 30 minutes, and I’ll post an update when finished.

Update 15:51: I’ve finished moving persephone to its new hardware, and it appears to be working properly. If you have any sites hosted on this webserver that are not working please contact technical support.

Update 16:05: It seems the new hardware I put persephone is having some problems of it’s own, so I’ll be moving it again. I apologize for the continued outages.

Update 16:54: New new persephone should be working properly now, we’re going to continue to monitor it carefully, and of course if you have any sites hosted on it that aren’t working please let us know.


8:23 am

Spunky Mail Issues

Posted (April 23rd, 2008 at 8:23 am PST) by Justin@DH

We are investigating what was causing customers some issues with pop and imap services this morning on the spunky cluster. At this time it seems to be working normally again but we are still testing and checking into the cause. Our apologies for the problems.

We believe we have found the problem server causing the downtime and it has been corrected. We’ve tested some email addresses on this cluster and they are showing pop, imap and smtp ok now. If you are still having issues please contact support with details.

Update April 28, 2008:

This was unfortunately not resolved as we are having problems with this cluster again today (authentication errors which from the reports I am seeing are effecting only pop3 as webmail is functioning). We have isolated it to the mail database and our database expert is correcting this now, we’ll add another update once we have more details from him.

Added information: we have a ‘master’ mail database and two clones so that we have plenty of redundancy and do not lose data. What happened here is that one of the clones is acting up and was refusing logins (I believe webmail is set to use the master by default which is fine and that would explain why webmail was working). We’re having normal mail use the master as well for now until we can work out the issue with the clone, so mail is working properly again while we make sure we have this fixed for you.

Quick note (04/26/2008):

We had a hiccup on a specific mail serer (just one out of the 9 on that cluster) which was giving similar errors to that resulting from the database server issues but we have fixed it and it is not actually related (just to alleviate concerns that the same problem was coming back).

Update: this was resolved per the more recent post on the subject and everything should be back under control.


4:37 pm

New network card for lovejoy

Posted (April 22nd, 2008 at 4:37 pm PST) by DreamHost

Lovejoy is getting a new network card to help with stability. The swap should go smoothly, but some downtime is necessary.

–Update: It took a bit longer than expected due to a complication. All fixed up now.


4:17 pm

Randy cluster networking problems

Posted (April 22nd, 2008 at 4:17 pm PST) by andrea

We’re currently experiencing networking problems on our ‘randy’ cluster (check the Account Status on the right upper corner of your control panel to see if you’re affected by this outage – if your email server is listed as randy, you may be). This is affecting web, mail, and mysql connections as well. We’re looking into this, and will update you as soon as possible. Very sorry about the inconvenience.

Sorry for the delay in an update, the networking issue has been corrected and since this the web servers working normally again. We will continue to monitor the situation, but if you are still experiencing issues related to this please contact support with details.


6:13 am

Migrating Agnus to new hardware

Posted (April 18th, 2008 at 6:13 am PST) by DreamHost

Agnus was running into networking issues all night and we are putting its hardware out of its misery. Currently one of our senior techs is working on failing it over to new hardware now. It shouldn’t take any longer than 30 to 45 minutes to complete and be up and running again. We’ll be sure to keep you up to date as things progress.

– Update 6:50 – Agnus is up and running with it’s new boots! Run Agnus, run!


7:04 pm

hupmobile moving to new hardware

Posted (April 8th, 2008 at 7:04 pm PST) by DreamHost

There’s an issue on hupmobile which is causing it to fail when booting. It’s being moved to new hardware shortly.


4:10 pm

Spunky / Spacey Mail Downtime

Posted (April 8th, 2008 at 4:10 pm PST) by brians

We are currently experiencing problems with the process that handles scheduled updates. This has caused a disruption in mail service. This will account for the password errors, imap connection errors, etc. that are being generated when attempting to access your mailbox. Newly created user aren’t being setup either. Our admins are diligently working to resolved the issue. We sincerely apologize for any inconvenience this may have caused. Please check back here for future updates.

UPDATE::2008-04-09 10:57 AM PDT

Our Admins are still investigating the cause of the issue. We truly appreciate your continued patience.

Update 04/9/08 5pm pacific: The problem is now resolved. If you’re still having problem with your email address, it’s unrelated, and we would be happy to help. Please contact support with details through the control panel.


9:10 am

Webserver “Hupmobile” being moved to new hardware

Posted (April 7th, 2008 at 9:10 am PST) by Justin@DH

The webserver hupmobile seems to have some bad memory so moving it to new hardware. Configs are already running and it should be backup in business completely within 30 minutes. Sorry for the downtime related to this webserver move.

— UPDATE Mon Apr 7 09:33:46 PDT 2008 —

All services are backup and running. If you notice further issues with hupmobile please contact support.


1:53 pm

30 min Blingy Downtime Tonight

Posted (April 6th, 2008 at 1:53 pm PST) by Josh

We may almost be in the clear with the problems we’ve been having with Blingy.

We now have a critical patch from our file server vendor which they believe will fix the crashing problems we’ve had. It takes 20 minutes to install, and unfortunately requires a reboot. We’ve decided to schedule this upgrade for 11:30pm PDT tonight, Sunday April 6th, 2008. That is pretty much the lowest usage time across all our users in a week, and we’ve decided we can’t wait another week before applying this patch in order to give more notice.

Our apologies for the very short notice! We received notice that the patch was ready this morning. Hopefully by midnight (PDT) tonight, Blingy will finally be happy again…

UPDATE 12:15AM PDT:

Still haven’t started the upgrade yet.. it took them a little longer to get the patch than expected, however, we should have it in the next 2 minutes, and then the upgrade will begin, probably around 12:30 now, and be done before 1am Monday morning PDT. Hang in there!

UPDATE 12:54AM PDT:

A few more bumps! Still no changes.. and now we’re afraid we’ll have to push off the upgrade to about 1:45AM (9:45AM GMT) it looks like.. hopefully all finished by 2:15AM PDT (10:15AM GMT), our apologies to blingy customers in Europe. :(

UPDATE 02:10AM PDT:

All done! The actual reboot only took about 3 minutes, and then there were some residual effects from the reboot on various web and mail servers for up to ten minutes after that. But things seem settled down now and the upgrade “took”. We all hope it did the trick. BIG TIME.


2:20 pm

Spunky Slowness and Downtime

Posted (April 3rd, 2008 at 2:20 pm PST) by DreamHost

Since the spunky cluster move some servers have been experiencing a higher than average load. This affected website load times, website downtime issues, webmail connections, email connections, and other services that reside on the individual servers in the spunky cluster. The cause has been traced to a cross connection saturation which links two areas of the data center. Capacity was doubled and the load on all spunky servers dropped back to normal levels. If you’re still experiencing problems please open a support ticket since this issue is now resolved.


 
© 1996-2008, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
11:25 am

WordPress 2.5 available in the One-Click Installer!

Posted (March 29th, 2008 at 11:25 am PST) by Jason C

Congrats to the WordPress team on shipping one of the biggest updates in recent memory, WordPress 2.5. They’ve completely refreshed the admin interface with the help of Happy Cog and while it might take a little getting used to, it certainly feels a lot nicer. More information on the release is available on the WordPress development blog and screenshots of the new interface can be found here.

The 2.5 update has been made available in the Advanced One-Click Installer and should be a fairly painless upgrade. If you’re itching to get your hands on it, go ahead and give it a spin as soon as you get a chance. We’ll be here if any questions come up, so feel free to file a ticket with our Support Team if you run into any major issues.

P.S. The update to the easy installer should be pushed thru between Monday, March 31st and Tuesday, April 1st. Sorry about the delay, but there are a few things that need to be done to make sure the upgrade goes smoothly for everyone.

P.P.S. If you wish to use the new plugin updater, you’ll need to temporarily disable “Extra Web Security” as there is a conflict with our current mod_security rules. Alternately, you can leave “Extra Web Security” turned on and update your plugins per their individual update instructions. This involves uploading them manually tho and can sometimes prove to be a bit of a hassle. We’ll try to have an update to our mod_security rules in the next few days tho and an update will be posted here when there’s a resolution to this issue.


10:37 pm

Filer problems with blingy cluster.

Posted (March 27th, 2008 at 10:37 pm PST) by jordan

We are currently having a problem with a filer which has crashed and is recovering at this time. While this is happening some customers in the blingy cluster will experience problems loading their websites/email. We apologize for the outage and service is expected to return to normal as soon as the filer recovers.

UPDATE 3:01:AM PDT

The filer has finished recovering and all services are back up and running. We are working with the filer vendor to find the source of the crash to prevent any further outages.

Update 24/03/08 10am: We’re working on the file server again to alleviate the load that’s causing problems with web, mail and mysql services. Sorry about that.

Update 27/03/08: We are doing emergency data moves to quell the stem of problems recently caused by your file server. During these moves, your data may be inaccessible. We are moving as we can off as fast as possible. Very sorry about the continued inconvenience!

Update 27/03/08 This series of moves has finished. We are going to keep an eye on things to see how much it helped and may have to do more moves tonight and tomorrow morning to get everything working smoothly again. This post will be updated with more information as soon as possible.

Update 29/03/08

We are continuing to move data off of the problematic file server but it’s a bit of a catch-22 because customers on that machine are continuing to add data at a very high rate. It filled up this morning for a while causing device full errors as well as mail problems and issues serving websites (when these fill up it causes problems across the board). To explain in more detail, when we move data it does not immediately disappear (there is a ‘snapshot’ created of the old data that remains in case there was a problem with the move – that ensures that we do not lose customer data but until the admin team can check the move to make sure it went through properly we cannot delete the old data). We just did some of that and have some breathing room again and of course more moves are still in progress but we are asking customer on this cluster to help us by holding up on any non-essential uploads of data for the next couple of days. As soon as we have a significant portion of the data removed the problematic file server will begin to function properly (and additional moves will go much more quickly and smoothly) but right now we’re having trouble moving data more quickly than it’s being added by people. If everyone could please limit uploads to absolutely essential data until we reach the turning point where everything is working this will be resolved much more quickly (in other words if for example you are setting up a repository of large files you’ll actually be better off waiting a couple of days and getting the all clear from us on this issue because you’ll be able to access that data reliably instead of cramming it on there now and slowing the recovery process).

In the meantime we’ll be doing everything we can to safely and quickly move data off and get things back to normal.

Added information: Some of the people recently moved to the new file server are seeing errors because the data did not get set up completely (loading the site will work but just show an empty index). The admin team has been running an rsync that will fully restore all data and should hopefully finish by 9 PM PST – once that is finished all site and email data will be available for those users.

Update 30/03/08

We’re still racing to keep ahead of new data being added so any help we can get on that front is greatly appreciated (we’re still asking for customers to limit uploads as much as possible to speed up the recovery process). Some customers who are being moved are seeing blank directories still but those are due to moves in progress and the data will be fully restored when those complete.

Update April 1, 2008

We seem to be ahead of the curve right now, we are moving data off the primary volume and on to a secondary one faster than new data is being uploaded. The volume hasn’t filled up completely in a few days. We are working closely with the technical support team to see how we can speed up the process further. Thank you for your patience.

Update, April 1, 2008

I apologize for the late update but we’ve been going over our options (while moving data of course). While we’re not seeing any real relief in terms of data uploads we do have some very large moves that are almost complete. Once those finish we can start deleting the data (for example one is around a half TB or around 500 GB which will be 4-5% of the total but it’s going to take until around Friday to delete it all, so we’re dealing with a ton of data). Tomorrow’s update should be earlier in the day and hopefully we’ll have some progress to reports from the large moves being complete.

Update, April 3, 2008

The data moves to other file servers has been running constantly, but last night and this morning some complications happened with the moves, requiring admin attention. To clear up some space there had to be a short interruption in file serving, this is now finished, space is available and the moves are continuing. The admins are fixing up the last of the web servers which were having issues after file serving was restored. Our apologies again for the continued issues.

Update, April 4, 2008

Today has been a pretty good day of progress. We were able to complete even more moves and free up more data from the file server. Moves have been going quicker and stability is dramatically improving. Monitoring of the servers and email in the blingy cluster today have shown a significant decrease in problems. Issues do still exist but the problem is noticeably getting better. We are also pleased to note that we have more storage that will be coming early next week. We believe that this will go a long way in helping us fix this major problem.

Update, April 4, 2008

Things are continuing to improve today – when I got in I was pleased to see that we had held firm and even gained a percent (the effected file server was down to 95% which is as low as I have seen it in the last week and since I have been working it has dropped to 94%). Performance should improve as we gain ground (this will speed up moving data off as well). This progress, along with the added storage space we are expecting early next week should hopefully allow us to restore service for our customers to normal.

5:45 PM PST : The moves we started just a while ago seem to be causing server problems, we’re looking into it and should have it resolved shortly (they were run just like the ones that had completed so we have to determine why these specifically caused an issue). Update: this resolved itself before we could detect the cause but we’re monitoring the situation to ensure that it’s not a recurring issue (we have no indication that it will be).

Update, April 8, 2008

Please see our other posting for details on the work we did on the effected file server:

http://www.dreamhoststatus.com/2008/04/06/30-min-blingy-downtime-tonight/

We are also continuing to offload data and are making good progress (it’s never as fast as we would like it to be of course). There’s excellent detail here in case you missed it:

http://blog.dreamhost.com/2008/04/07/another-anatomy/

which chronicles the situation and fills you in pretty much up to today. We’re seeing the data dip to 90% so we’re hoping to have it down in the 80’s by the end of the week (every percent we gain helps and as performance improves we can speed up the rate of moving but we’re still looking to hit critical mass where you get the proper level of performance).

Update, April 9, 2008

As we had hoped progress is speeding up as we free up more space – while the file server is showing 95% usage, around 11% of that is data that has already been moved and is no longer in use. Due to a software issue we haven’t been able to remove it yet (the admin team is working on the best way to execute that), but once that is gone we should be around 85% usage which is another large step forward.

In terms of effect, I have already seen improvement in site function for many customers as well as greatly increased speed in moving chunks of data off as well as receiving reports that mail is functioning quite a bit better. That said this issue remains at the High severity rating and in unresolved status as we have not reached a normal level of service. I can’t stress enough how sorry I am that our customers have had to put up with this but I thank those of you who have stuck with us (check the newsletter for details on what we’re doing for Blingy customers) and look forward to providing you with the level of service we strive for at DreamHost.

Update, April 9th, 2008 21:59 PDT

Unfortunately, we need to unmount the volume again to kill these snapshots before they leave us with 0 bytes of free space. In 2 hours (midnight) I will be taking the problematic volume offline to delete the phantom snapshots. Total downtime will be between 10 and 30 minutes. Sorry for the short notice and additional outage!

Update, April 10, 2008

Well the snapshot mentioned yesterday is gone and we’re actually at 83% used today which is below where we were hoping to see marked improvement (85%). Of course we’re still moving data off (which increases the usage on the file server) so that won’t fully translate to customer usage improvement but it should be quite a bit better and keep improving until we stop moving data.

Update, April 14, 2008

Okay, we’re finally getting ready to mark this as resolved.. things have seemed pretty much okay for a while now. But, just to be sure, we’re dropping the severity to Medium for now and leaving it as unresolved.

Update, April 17, 2008

We’re still hearing some reports of site slowness – we were able to resolve an issue causing high loads today which should help but we’re not going to consider this resolved until everyone is receiving good service.


1:12 pm

Problem with central services (panel,webmail, etc..) and spunky cluster

Posted (March 25th, 2008 at 1:12 pm PST) by jordan

We are currently experiencing problems with one of our internal firewall machines that is causing problems with our central services connecting to our customer database. This problem is expected to be fixed within 20 minutes, we apologize for the trouble and will update this post as soon as it is resolved or more information becomes available as to the cause.

UPDATE 2:40 PM PDT **

Problems with central services have been resolved. Panel , webmail and other central services should be working as usual. There are still a few problems with the spunky clusters that may be causing outages for customers websites/ftp. These problems are also being fixed at this time and this post will be updated when those are fixed.

UPDATE 4:39 PM PDT **
All of the networking problems and spunky cluster problems have been resolved. We apologize again for the outage and if you are still having trouble feel free to contact support with the service you are having problems with.

UPDATE 5:11 PM PDT **
There have been reports of problems with the spunky cluster , we will re-open this status post until we have more information to what caused this problem. If you are on the spunky cluster and are experiencing downtime it is still due to the networking issue.


1:00 pm

Migrating bellflower to new hardware

Posted (March 22nd, 2008 at 1:00 pm PST) by Jason C

It looks like bellflower was a casualty from last night’s move. Currently one of our senior techs is working on failing it over to new hardware now. It shouldn’t take any longer than 30 to 45 minutes to complete the move (unless something goes horribly awry) and make sure everything is plugging along again. We’ll be sure to keep you up to date as things persist.

All fixed – Bellflower is on much newer/better hardware now too as that was one of our older shared hosting machines (just a fortunate positive side effect of the failover).


7:50 pm

Spunky and Central Move

Posted (March 21st, 2008 at 7:50 pm PST) by Terri

We’re about to begin to move the spunky cluster and our central machines. We’ll be taking down servers soon, but our status site, of course, will remain up and running.

Again, we’ll try to keep this move quick and painless as we possibly can, and we apologize for the inconvenience.

For more details on this, see the previous post here:
http://www.dreamhoststatus.com/2008/03/14/central-services-and-spunky-cluster-move/

UPDATE — Sat Mar 22 00:12:05 PDT 2008 —

Most central functions are back up. Dreamhost.com and panel.dreamhost.com are working again. We are currently working on webmail. The spunky cluster move is still underway. We’ll post more here later.

UPDATE — Sat Mar 22 02:45:26 PDT 2008 —

Webmail is up! 99% of all central services are functioning now. Spunky is 80% racked, more info on that to follow.

UPDATE — Sat Mar 22 03:40:27 PDT 2008 —

Spunky is coming up. We’re spotchecking and fixing troubled file, web, and mail servers. We’ll sound an all when all is clear… We will likely be firefighting for the next few hours.

UPDATE — Sat Mar 22 06:01:28 PDT 2008 —

Spunky is still coming up, we’re dealing with a couple of machines that didn’t survive the move and some more cleanup stuff. We’ll keep you posted as things progress!

UPDATE — Sat Mar 22 07:04:35 PDT 2008 —

We have noticed a problem with some of our networks. They seemingly did not make the jump to the new router and we are looking into that. Any websites with IP addresses that look like either 64.111.xxx.xxx or 208.113.xxx.xxx are down to the outside world right now. The machines themselves are up so as soon as the networks are working again they’ll begin working immediately.

UPDATE — Sat Mar 22 09:55:54 PDT 2008 —

The networking problem has been resolved with our network provider so as of right now all services should be up and running. If anything is not working, please open a new support message from our web panel.


4:39 pm

Janky Cluster Filer Problems

Posted (March 21st, 2008 at 4:39 pm PST) by DreamHost

We’ve been working on an issue with the Drooler file server which is in the Janky cluster. We were able to correct some initial problems we saw that was causing problems such as high load, slow email, slow websites, sites timing out, and email connectivity issues. Some of you might have received an email saying the problem was fixed, and it was, however the symptoms have returned. We have our Admin Team currently working to resolve this issue as quickly as possible. Once this issue is corrected we’ll update this post with more information.

Servers that can be affected from this are listed below:

adonis alboz anton anwaltskanzlei apollo aries atlas azcuehnos baachus barefoottours batserver bogus charon charybdis chronos dione dionysus djm electra erebus erida fresh gladys gorgon helios hera hypnos icarus iris jaky janky-vserver01 janky-vserver02 janky-vserver03 janky-vserver04 janky-vserver06 janky-vserver07 janky-vserver08 janky-vserver09 janky-vserver10 janky-vserver11 jove kermit kiru kratos kztixo medusa mf1 mf2 midas morejewel morpheus mytimeofday nemesis ninja odysseus oriza osiris pan pegasus persephone phi privateserver rhea shyfry sisyphus somaticvision statler tartarus tethys thanatos thetis triton tyche vivenet warppipevps zeus

Update: Drooler is back up, and we will be monitoring the situation closely in case any further problems arise.

Update 4/8/08 4pm : We have added a new file server a couple of days ago and have another one on order. We continue spreading user data across the existing file servers, and are looking into possible user issue that are causing the servers to be unstable. Sorry about the inconvenience.


2:22 pm

A Fistful Of One-Click Upgrades

Posted (March 21st, 2008 at 2:22 pm PST) by Jason C

After taking way longer than we should have to get these out of the way, we’re happy to announce the following one-click upgrades as available in our One-Click Installer:

  • ZenCart 1.3.8
  • Pligg 9.9.0
  • Moodle 1.9
  • Joomla 1.5.1
  • MediaWiki 1.12.0

If you are running previous versions of any of this software on your sites, please consider taking a few minutes and running the upgrader. It doesn’t take very long to do and staying on top of available upgrades is the best way to make sure your site stays stable and secure.

As usual, if you notice any errors caused by the One-Click Installer, you can let us know by creating a new ticket with DreamHost Support.

Happy upgrading!

P.S. There will also be an upgrade to WordPress as soon as version 2.5 becomes available in about a week or so. Currently it’s testing and getting a few final touches put into place — but I thought it was worth mentioning as there will be some big changes with the admin interface, themes and plugins. To make sure your WordPress based site is ready, you should give this handy rundown by Lorelle VanFossen a once over. That should help cut down on any snags that might be caused by the upgrade.

P.P.S. If you are on Twitter, you might want to consider following dhstatus so you can get DreamHost Status blog updates via SMS, Jabber or AIM.


8:13 am

Networking problems in one of our datacenters

Posted (March 19th, 2008 at 8:13 am PST) by andrea

We are currently experiencing networking issues at one of our data centers, which is causing downtime on many clusters, and dedicated servers, affecting email and websites. Our networking admins are on the case, and will have this resolved as soon as possible.

Please check back here for updates. We apologize for all inconvenience this has caused.

Update: as of 10:30am Pacific Time, this outage is resolved. If you’re still seeing problems with your domains, please let support know, so we can look into it for you.


5:26 pm

Phil isn’t feeling very well

Posted (March 18th, 2008 at 5:26 pm PST) by Jason C

Our admin team is currently investigating an issue with the MySQL server phil on the blingy cluster. Right now, any site that utilizes phil will be running into connectivity issues and new databases cannot be created on the server. Sorry to everyone that is inconvenienced by this service being down.

Hopefully we’ll be able to get a lock on this soon and sort this out. Please continue to stay here for all the latest updates on phil’s condition.

Update Tue Mar 18 18:52:04 PDT 2008: phil is back online.


3:12 am

Problem with down websites

Posted (March 18th, 2008 at 3:12 am PST) by jordan

We are having some problems that are not cluster specific regarding broken apache configurations. The problem is being fixed right away and we apologize for the outage and will resolve this post when all sites are back up and running.

Update: As of 9am Pacific Time, the problem has been resolved . Sorry about this inconvenience this downtime has caused you.


 
© 1996-2008, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
2:05 pm

FastCGI Environment Changes for Rails – Updated

Posted (March 14th, 2008 at 2:05 pm PST) by DreamHost

Recently we’ve made changes to FastCGI to disallow any environment except for Production when using Rails. This was brought on by some issues with the Development environment causing the quality of service to degrade. If you need to run your application in Development mode, please use your local machine for this. If you have a live application that was running in Development mode and now it doesn’t work, this is most likely the cause. You can correct this by making sure you have setup your database.yml file to have the proper MySQL connection information for Production, and that you’ve changed your environment.rb file to reflect these changes.

Update: I’m sorry, my first post isn’t entirely accurate. We are not disallowing any environment except for Production. We’re simply defaulting Rails installations to Production instead of their old default which was Development. This doesn’t mean we allow development on our servers, and in fact, we frown upon any development on our production servers because of the risk involved. I understand some people are concerned about staging, and this should be fine, so long as you use CGI instead of FastCGI.


12:00 am

Central Services and Spunky Cluster Move

Posted (March 14th, 2008 at 12:00 am PST) by micah

Due to continued space and power constraints in our primary data center, we will be moving all central DreamHost functions as well as the “spunky” cluster to one of our newer data centers. This move will begin Friday, March 21st, at 9PM PDT, and is expected to last up to 12 hours, until Saturday, March 22nd, 9AM PDT.

Central services include panel.dreamhost.com, dreamhost.com, dreambook.com, files.dreamhost.com, media.dreamhost.com, wiki.dreamhost.com, discussion.dreamhost.com, basically everything but secondary DNS and this status blog. Primary and tertiary DNS will be going down as well, but will have a more limited downtime window of only 2-3 hours. For spunky, all web servers, mail servers, file servers, and MySQL servers in the spunky cluster will be unreachable. The list of affected spunky web servers is here:

abbazaba absinthe achilles adams admiralty aero alameda almalibre alondra altoid alvarado archibald arrow atlantic avalon aviation babyruth badillo balu bandini barranca barry baseline bazooka beaudry bellflower beverly bixel blowpop bonbon bottlecap bounty bourbon boyd broadway brownie bubbleyum bundy burton butler butterfinger cahuenga caramello casamento catalina centinela century cerritos cheeto cheezit chex chiclet cienega cognac colbert compton cookie cordial crackerjack crenshaw crunch cupcake dingdong doheny dorito drumstick dumdum edendale euclid fairfax fc23 federal fig fireballs firestone fletcher flower foothill formosa frito frontseat fruitland fudge fundip garland garth garvey gobstopper goober good gower grand granola grappa gumdrop gummi haven heath heaven highland hillhurst hillmont hoho holt hoover hope hops hyperion jalapa jawbreaker jefferson jerky juniormint jurupa kidsgames kiki kitkat krackel lankershim larchmont laurel lemonhead licorice lifesaver limbo-spunky3 limbo-spunky4 locust look lucas malt manchester mariposa mayo melrose mento milliken mills monroe moorpark mounds munchies nash necco nerds nilla nougat nutrageous oakdale ocean olive olympic ord oreo ouzo overland oxnard paramount payday pernod pez pico planters plenty pocky popcorn poprock powerbar pretzel price pringle rampart razzle redhot redondo reeses rexford ritz rockyroad rodeo rollo roscoe rosecrans rossmore rum runts sake saltine sanborn sawtelle schnapps scotch selma sepulveda shocktart skittle skor skybar slauson smores snicker snocap soto sourpatch spree starburst sugardaddy sunset taffy tampa temple tequila toblerone toffee tootsie topanga torrance truffle twinkie twix twizzler union vanowen vermouth vine virgil warhead washington western whittier whopper wilmington wilshire york zagnut zuma

We realize this is a very large move and we will have a dozen techs onhand to supervise. My apologies for not posting this earlier.

Happy DreamHost Mover Team!


11:54 am

LAX Networking upgrade

Posted (March 10th, 2008 at 11:54 am PST) by DreamHost

On Wednesday, March 12, starting at 10PM we need to do a quick network migration. We are moving from one router to a newer, larger router. This will hopefully solve a problem we are having with load balancing across our links back to our downtown datacenter. Cisco isn’t sure what is causing it, and we have no free capacity to do the troubleshooting they need to properly diagnose the problem.

The outage should be around 30 minutes, and will affect dedicated servers, looney, swarthy, spacey, frisky, and blingy. This is going to run concurrently with the Blingy file server upgrade.

Update 11:15PM Pacific: As of right now, I believe we are back in business. The blingy cluster took a bit longer to show OK due to the upgrade concurrently happening. Please let Technical Support know if after right now you are still having problems!


9:52 am

Blingy file server upgrade

Posted (March 10th, 2008 at 9:52 am PST) by DreamHost

On Wednesday night, March 12, 2008 starting at 10:00pm Pacific time, we will be performing a major OS upgrade on the file server that handles the entire blingy cluster. This upgrade is at the direct recommendation of BlueArc to hopefully resolve periodic slowdown issues and other quirks of the system. The estimated downtime by the BlueArc technician is 1 hour.

Quick update: This is the first upgrade we couldn’t do live behind the scenes. We have already upgraded once to the latest version that could be applied live to the clustered heads.

Update 11:15PM Pacific time: The BlueArc technicians are still working on the upgrade. It seems they are nearly completed, and one of the redundant pair of heads seems to be serving data right now. Stay tuned for a final update.

In other news, while BlueArc and I were standing there next to the file server discussing the problems that have been plaguing us at 7-10pm, it happened! The unit crashed and completely stopped serving data. I left them to gather their crash data and continue the upgrade process.

update 11:58PM Pacific: Everything looks to be better. Please contact technical support if you are having any problems!


10:38 pm

Problems with blingy cluster’s filer

Posted (March 7th, 2008 at 10:38 pm PST) by jordan

We are currently having some problems with one of our blingy clusters filers. It is being looked into and fixed at this time. Please check back as we will post more information when it becomes available.

UPDATE 4:00AM PDT
The problem with the filer has been resolved but there are still issues regarding sites down that are being addressed. Please see post above for info.

UDPAT


8:47 pm

Dreamhost firewall error (Resolving)

Posted (March 7th, 2008 at 8:47 pm PST) by DreamHost

Due to a typing error on our primary router while trying to block a denial of service attack, DreamHost is currently offline. This includes all email, web hosting, etc. A technician is about 15 minutes out from our datacenter to undo this mistake.

I apologize for this mistake. I was intending to be editing our non-live access-list, and edited the live one by mistake.

Update 9:54 PM Pacific: We have unblocked the firewall and our routing is coming back online. Sites should start working again as soon as our router finishes loading its tables. I use “resolving” because it is still processing the routes.


5:32 pm

Problems with bowser

Posted (March 6th, 2008 at 5:32 pm PST) by DreamHost

This server isn’t coming up when rebooted. It’s now being moved to a new machine. Should be back up shortly.

Update: this was completed but the post wasn’t resolved. Sorry about that.


8:02 am

Spunky File Server Issue

Posted (March 5th, 2008 at 8:02 am PST) by Justin@DH

We are currently experiencing problems with one of the file servers in the “spunky” cluster. An admin is already on the way to the data center to investigate. When the issue is resolved or we have further information an update will be posted. Our apologies for the issues, we are working to get this fixed up as soon as possible.

Update: 8:15am pacific: one of the fibrechannel interface cards died and had to be replaced. We’ve completed this process and the filer is back online now. Services should be back up as usual!


5:01 pm

Webserver korben being moved to new hardware

Posted (March 4th, 2008 at 5:01 pm PST) by DreamHost

The webserver korben has had one of its nics fail, so we are moving it to new hardware. This process should take about thirty minutes, during which time all sites hosted on korben will be down.

Update: this has been finished, please contact technical support if any of your sites hosted on korben are still having problems.


9:25 pm

Blingy File Server Issues

Posted (March 3rd, 2008 at 9:25 pm PST) by glen

A file server on the Blingy cluster is getting backed up with IO processes. This is leading to sky high loads and slowness across the cluster. An admin is looking into this as well as updating a trouble ticket with the file servers vendor.

Affected Servers:
agnus altair birdo bonk boo bowser clank cloud conker daxter dct ermac flamenco fox ganon glassjoe jackie jak jax kirby kyrie lara link luigi mario megaman nariko nero ness nukem peach pikachu pit popinski ratchet scorpion shinobi snake spyro strider subzero tails toadstool voldo yoshi zelda


 
© 1996-2008, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
4:28 am

Randy cluster file server issue

Posted (March 1st, 2008 at 4:28 am PST) by DreamHost

One of our file servers on randy cluster is having issues this morning. Our admins are on it and it’s getting fixed right now.

This affects the following web servers:

attila baht bob centipede contra cornelius cortes deblume defender digdug dinero ellipsis gage galaga go guilder herod ikari io klax knocker korben kroner lira mark morgan napoleon nimitz ninja orbital patton peso pizarro quid rupee rygar scagnetti shekel slimy tetris ugh venice vex vito wasabi yen yoda alexander bam boppity breakout caesar crack cruisin dollar euro forint franc grant happy ike ivan janice jezebel leeloo lei limbo-randy3 lucky paperboy pound qbert schilling scipio zorg

Update 5am: Should be fixed now.  The file server is back up and I’m just cleaning up peon loads and such.  If you are still having issues, feel free to write in about it.


2:06 pm

Mysql server deputydawg going down for hardware replacement

Posted (February 28th, 2008 at 2:06 pm PST) by DreamHost

The mysql server deputydawg is having some problems with its raid array, and needs to be replaced. There has been no data loss. This should take about thirty minutes, during which time any databases hosted on it will be inaccessible.

Updated 2:44pm, this is done, deputydawg is on new hardware and serving its databases.


4:14 pm

Problems with Janky cluster

Posted (February 25th, 2008 at 4:14 pm PST) by andrea

If you’re hosting on our janky cluster (see under Account Status/Your Email Server in the control panel), you may have been experiencing slowness with your site or difficulties checking email this morning. The reason for this was that one of the file servers was having connection issues. We found and already disabled a customer that we believe was causing a huge strain, and are looking for others. The file server is looking much better, please try checking email of check your sites again.

Very sorry about the inconvenience this has been causing you. If you’re still seeing problems, it may be unrelated, so please let us know (with details) by contacting us at https://panel.dreamhost.com/index.cgi?tree=support.msg&

Update: 6:00pm (PST): These issues have been resolved for the last few hours, but we wanted to make sure before posting and update. All of the mail login and site speed tests we’ve run are going through fine and things seem back to normal. If you’re still having problems with any sites or email hosted on the ‘janky’ cluster though, please contact support right away.


12:47 pm

Mail delay on junk filtered domains

Posted (February 25th, 2008 at 12:47 pm PST) by andrea

Most of our customers using junk filters on their domain(s) are experiencing delay in receiving mail. This is affecting all mail clusters, but only the domains that have junk filtering enabled through the control panel. Mail is slowly going through, but it’s still very backed up, so it will take a while to catch up. We added more machines to alleviate the problem.

Very sorry about the inconvenience! We won’t mark this post resolved until the clog is all gone, so please check back for updates.

If your domain is having delay issues despite it not being filtered, please let us know at https://panel.dreamhost.com/index.cgi?tree=support.msg& . We’ll need details like the address affected, the full headers of the delayed message, or the sender and recipient and when the message was sent, so we can look into it for you.


9:59 am

Web Panel Cronjob Changes

Posted (February 21st, 2008 at 9:59 am PST) by kitchen

After much research we have noticed that most people who set up cronjobs either through the web panel or through the crontab command line utility never read the emails it sends them, and, in fact, most probably don’t even know they are receiving them. These unwanted and unnoticed emails use up considerable amounts of space on our file servers, as well as causing performance issues if several of the maildirs they are delivered to fill up to capacity, and can cause ssh logins to become very slow, as most shells will check for new email for you at login, and checking 300000 emails can take some time.

Because of this, we’re going to make some changes to the web panel to support changing the email address a cronjob’s output is delivered to. Yes, huge feature, we know :) The reason it’s being announced here rather than just sneaking quietly into everyone’s web panels is because of the implications. Since we do not have existing data for where emails are being sent, we’ve decided that having it default to being disabled would be the most logical choice.

What does that mean for you? After next Tuesday (February 26th) when you edit a cronjob in the web panel, any other cronjobs associated with that user will get their email delivery disabled until you modify the email delivery settings for those jobs. To fix this, simply modify the other jobs and change their email delivery settings!

If you have any questions about this change, please feel free to contact our technical support department and they’ll be happy to help you out!

Thanks a bunch!

Update 2008-02-26: this is now live. Please let us know if you have any questions or problems by sending in a support ticket!

– The Happy DreamHost Make Filers Happy Team.


6:57 am

DreamHost Wiki and WebFTP sites are down. FIXED!

Posted (February 15th, 2008 at 6:57 am PST) by David

We apologize for the inconvenience. Unfortunately there’s a problem that’s causing both our Wiki and WebFTP service to give an error. Our admin team has been alerted and is currently working to resolve the problem as quickly as possible. No need to submit any support tickets for either of those problems at the moment.

If you can use a stand-alone FTP client in the mean time I’d recommend doing that until we can get the WebFTP service back on-line again.

Thanks for your patience while we work to resolve these issues.

Your-early-morning-“we’re fixing it now”-team.

UPDATE: 7:11 PST. They’re both fixed now! If you have any problems with either feel free to submit support requests again via the normal channels.

Your-early-morning-“it’s fixed now”-team. ;-)


8:51 am

Janky cluster file server issue

Posted (February 14th, 2008 at 8:51 am PST) by Justin@DH

One of the file servers on the janky cluster is having issues this morning, an admin has already been notified and the situation is being investigated. We are working to get this resolved and the file server back up as soon as possible. Our apologies for the downtime related to this issue.

Update: It’s back up and running. It seems to have stopped functioning while re-building disks, but we will need to do more diagnostics to be sure of that assessment.


8:13 am

Mail originating from web servers delayed.

Posted (February 11th, 2008 at 8:13 am PST) by David

Due to an intermittent problem with the mail server that processes outgoing messages from the web servers some messages may be significantly delayed in their delivery. We are doing what we can to resolve this situation and try to prevent it from happening again in the future. We apologize for any inconvenience this may cause.

Update 8am:  This is now fixed.  Our outgoing mail servers from web servers had managed to accrue over 1.25 million emails in their queue from web servers and this was keeping them from sending email.  The only thing to do to get them to work again was to turn their ability to accept incoming email off for a bit (yesterday) so that they had time to send out the massive queue they had.  Yesterday at about 3pm I turned on incoming email again, which made all the web servers send them their entire queue (about another 800,000 emails).  However, as of this morning the queue has about 400 messages in it so we are all good again.  In part to make this work better I’ve tweaked the settings in a way that allowed them to send over 2 million emails in the space of 24 hours, which should keep this from happening again.


9:29 pm

Randy Move going ahead

Posted (February 8th, 2008 at 9:29 pm PST) by DreamHost

Just a reminder that we are moving the randy cluster starting at 10pm. All services associated with it will be down until we are done.

Update: We’ve finished moving all the hardware, and have things powered on. We are now cleaning up some lingering issues, so please refrain from contacting support until we post an all clear. This applies to some issues that happened to spunky cluster as well.

Update: For the most part everything on the randy cluster should now be back to normal. If you are still having issues, please contact support!


9:19 am

File Server Issue

Posted (February 8th, 2008 at 9:19 am PST) by Terri

One of our file servers is experiencing difficulties that we are currently working on diagnosing. This will cause load and downtime issues for users on these servers:

abbazaba absinthe achilles adams admiralty aero alameda almalibre alondra altoid alvarado archibald arrow atlantic avalon aviation babyruth badillo balu bandini barranca barry baseline bazooka beaudry bellflower beverly bixel blowpop bonbon bottlecap bounty bourbon boyd broadway brownie bubbleyum bundy burton butler butterfinger cahuenga caramello casamento catalina centinela century cerritos cheeto cheezit chex chiclet cienega cognac colbert compton cookie cordial crackerjack crenshaw crunch cupcake dingdong doheny dorito drumstick dumdum edendale euclid fairfax fc23 federal fig fireballs firestone fletcher flower foothill formosa frito frontseat fruitland fudge fundip garland garth garvey gobstopper goober good gower grand granola grappa gumdrop gummi haven heath heaven highland hillhurst hillmont hoho holt hoover hope hops hyperion jalapa jawbreaker jefferson jerky juniormint jurupa kidsgames kiki kitkat krackel lankershim larchmont laurel lemonhead licorice lifesaver limbo-spunky3 limbo-spunky4 locust look lucas malt manchester mariposa mayo melrose mento milliken mills monroe moorpark mounds munchies nash necco nerds nilla nougat nutrageous oakdale ocean olive olympic ord oreo ouzo overland oxnard paramount payday pernod pez pico planters plenty pocky popcorn poprock powerbar pretzel price pringle rampart razzle redhot redondo reeses rexford ritz rockyroad rodeo rollo roscoe rosecrans rossmore rum runts sake saltine sanborn sawtelle schnapps scotch selma sepulveda shocktart skittle skor skybar slauson smores snicker snocap soto sourpatch spree starburst sugardaddy sunset taffy tampa temple tequila toblerone toffee tootsie topanga torrance truffle twinkie twix twizzler union vanowen vermouth vine virgil warhead washington western whittier whopper wilmington wilshire york zagnut zuma

If you are on these servers, this will may also cause email services to be unavailable. We apologize for this, and we will post more information as it becomes available. Thank you for your patience.

Update 9:50 a.m. PDT: The file server had hung on a thread and decided to throw a kernel panic. Right now it’s saving crash data and then should boot up. After it boots up I’ll need to soothe the servers that are hurting because of the broken mount. We should be back up and running within a half an hour, but I’ll post again if something else goes wrong.

Update 9:56 a.m. PDT: The file server was still not happy and died again. We are still working on resuscitating it.

Update 10:08 a.m. PDT: We found the problem. A bad drive was completely hosing the file server. We replaced the drive and it booted just fine. I’ll be massaging servers that still hurt now.

Update 12:38 p.m. PDT: The rest of the servers that were affected by this have been fixed. If you continue to have issues, please contact support.


 
© 1996-2008, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
1:50 pm

Pre-randy cluster move setup

Posted (February 1st, 2008 at 1:50 pm PST) by DreamHost

On Tuesday, February 5th at 10:30PM pacific time, we need to swap two volumes on two filers. These filers have both Randy and Spunky mail cluster data on them, and they need to be isolated.

This whole process should take no longer than one hour. During which time your email will queue on remote servers and be delivered when we are finished, but you will have no access to your files or websites.

–UPDATE– Mon Feb 4 11:23:25 PST 2008

Our apologies for the vague posting. To clarify, this will affect only 1% of the customers in either the Randy or Spunky clusters. Unfortunately we can’t list exactly which customers will be affected but it is extremely limited in scope.

–UPDATE– Wed Feb 6 03:15 AM PST 2008

That did not go as smoothly as we would have liked. Most servers are back online now, and service should be resumed for most people as of one hour ago. It appears that most of the peons did not give up their old mount points as easily as we usually see them. We also have never attempted this style of a move during live service. On Friday night it should not be affected nearly as much, since everything will get clean reboots immediately after we move the stuff around.


12:32 pm

Network troubleshooting tonight.

Posted (January 31st, 2008 at 12:32 pm PST) by jeremy

To help debug some issues we’ve been having with an Etherchannel trunk in one of our facilities, Cisco has asked us to bring one of the members of the trunk offline tonight at 9pm Pacific time. Even with the 1 member offline, we should have plenty of spare capacity to handle all of the traffic, so we’re expecting little to no impact on customer sites.


9:52 am

Web server “electra” is having a bad day! – UPDATED(2x!)-

Posted (January 31st, 2008 at 9:52 am PST) by David

I apologize for any inconvenience this is causing. The server “electra” is locked up and won’t even power cycle at the moment. We’ve dispatched a technician to fix it but it could take 30 minutes or so. We’ll post an update as soon as the problem is corrected.

This will effect web services if you’re domains are hosted on this server, but e-mail and database services will be unaffected.

The DreamHost-We’re-Fixing-it-now Team.

UPDATE 1/31/2008 10:32 PDT: It’s back up again! Yeah! Sorry about that.

Update 2/8/2008 5:!7AM Pacific: Looks like it was getting hammered with a denial of service attack this morning. I have hopefully disabled the domain which was the target of the attack, but only time will tell. The apache service has a new IP assigned to it, and DNS is updating globally.


4:08 pm

Dhalsim being moved to new hardware

Posted (January 30th, 2008 at 4:08 pm PST) by DreamHost

The webserver dhalsim is being moved to new hardware due to the old stuff failing. This should take about thirty minutes, I’ll post an update when finished.

This is finished and new dhalsim is up and running. If you have any sites on it that are not working properly please contact tech support.


2:30 pm

Last Chance to donate to The Life Link and VHL

Posted (January 30th, 2008 at 2:30 pm PST) by Josh

Hey!

Tomorrow is your last chance to donate to The Life Link (a homeless shelter in Santa Fe) and/or VHL Family Alliance (a Von Hippel-Lindau disease charity) and get your donations matched by us!

So far we’ve got 18 donations for a total of $614 to The Life Link and SIX donations for $121.63 to VHL Family Alliance.

We can do BETTAH!


3:34 am

Quick Web Panel Maintenance 2008-01-30!

Posted (January 29th, 2008 at 3:34 am PST) by kitchen

We will be taking the Web Panel down tomorrow night for about 20 minutes for some quick maintenance. This will start at 1:00am Pacific on Wednesday January 30th. This will not affect your hosting, databases, email, webmail, or anything else. Only the web panel!

We will post here again when we begin, and provide updates as things progress.

Update 1am pacific: the web panel is going down now. We expect the downtime to be about 20 minutes. Thank you for your patience!

Update 1:44am pacific: the web panel is back up and running! It was actually only down for about 20 minutes, there were a couple of quick things we had to do before taking it down.

Please contact support if you are having any difficulties!

Thanks!

-The Happy DreamHost Late Night Panel Maintenance Robot


4:50 am

spunky filer offline

Posted (January 28th, 2008 at 4:50 am PST) by DreamHost

One of our filers in the “spunky” cluster seems to have crashed and is not coming back up remotely. Our technicians are on their way in to the office to attempt a repair at this moment. The current ETA is around one hour.

Edit: 6:18AM Pacific – The filer has been fixed, and we believe to have most web server issues resolved. Please contact support if you are having trouble!


7:31 pm

fidget hard drive failure

Posted (January 26th, 2008 at 7:31 pm PST) by kitchen

Fidget’s boot drive has died and will not boot, so we’re going to move it over to some new hardware. The process should take about 30-45 minutes and we will post an update here when it is finished.

Update: 7:44pm pacific: fidget has been failed over to new hardware and services are coming up. We will post another update when things are 100% back to normal!

Update: 8:00pm pacific: things should be back to normal on fidget. Please contact support if you are still having issues!


9:45 am

Randy cluster move

Posted (January 26th, 2008 at 9:45 am PST) by DreamHost

Due to continued space and power constraints in our primary data center, we will be moving the “randy” cluster to one of our newer data centers. This move will begin Friday, February 8, at 10PM PST, and is expected to last up to 8 hours, until Saturday, February 9, 6AM PST. All web servers, mail servers, file servers, and MySQL servers in the randy cluster will be unreachable during this time.

The following web servers will be off line during this move:

alexander arkanoid attila baht bam bob bomberman breakout caesar centipede contra cornelius cortes crack cruisin cssmania deblume defender digdug dinar dinero dollar donkeykong ellipsis euro fidget forint franc gage galaga gap go grant guilder happy herod ikari ike io ivan jezebel joust karnov katamari khan klax korben kroner lamlam leeloo lei limbo-randy limbo-randy2 limbo-randy3 lira lucky mark marvin morgan napoleon nexus nimitz orbital pacman paperboy patton peso pizarro pound qbert quid rhod ruble rupee rygar scagnetti schilling scipio shekel slaughter slimy tetris ugh ultima venice vex vito wasabi yen yoda zorg

If your site is on any of these servers this status post applies to you. Also, if you check the “Account Status” button on the upper right hand of the webpanel ( https://panel.dreamhost.com ) you should see your cluster under the section “Your Email Server:”. We are also sending out an email announcement about this downtime window to affected customers.

We do apologize for this inconvenience. Rest assured we will do everything possible to keep downtime to a minimum.

If you have any questions regarding the server move, please contact our support team.


8:08 pm

LAX data center connection issues

Posted (January 23rd, 2008 at 8:08 pm PST) by glen

We are currently experiencing connectivity issues to our LAX data center.

All servers in the following clusters would be involved:
Blingy, Frisky, Looney, Spacey, Swarthy

Updates to come.

Update: All Servers seem to actually be up, but there is heavy packet loss at the distribution switch making them seem down to the outside.

Update: Packet loss has stopped. Our admins do have significant network upgrades planned shortly as a long term fix connectivity issues like this. If you have any further downtime , let support know via our panel form.

Update: Spoke too soon, Packet Loss is back . Our admin team is working feverishly on this.

Update – Wed Jan 23 21:34:52 PST 2008 – After a reload the switch is working properly again. We are already in the process of a major upgrade to this particular part of the network to resolve this issue. The actual swap-out of equipment will happen as soon as possible. Until that time we will be taking steps to keep the current switch operating properly.


 
© 1996-2008, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
6:21 pm

Billing Issues Update

Posted (January 20th, 2008 at 6:21 pm PST) by Dallas

We believe all of the billing errors to have been corrected as of the morning of Friday, January 18, 2008. If your credit card was wrongly charged, the charges have been refunded. We saw some reports that an expected refund did not appear, but upon investigation the refund had indeed been processed on our side and had just not yet made its way to the bank. We have not seen any cases where a refund has not been issued. It can take several days for refunds to appear back on your credit card or in your bank account and that is unfortunately entirely out of our hands.

And remember…

If this/these erroneous charge(s) by us resulted in you having any sort of overdraft/bounced check/nsf fee from your financial institution, please contact our support team from the web panel. We’d just like to request that you include a copy of your statement with the necessary info showing the fees. It can be either a paper statement or a print out of your online statement, or even a screenshot of your online statement and it can be scanned and attached to your support message via our support form or faxed to us at 714-990-2600. If you fax it, please be sure to write your domain name or DreamHost account number on the fax. When we get this, we will put money on your credit card equal to the amount your bank charged you, as well as give you a DreamHost account credit for the same amount on top of that.

We will do our best to correct any problems stemming from this error.

In case you have not seen it, we have posted information about this situation in three separate blog posts on our official blog…
http://blog.dreamhost.com/2008/01/15/um-whoops/
http://blog.dreamhost.com/2008/01/16/the-aftermath/
http://blog.dreamhost.com/2008/01/17/the-final-update/


12:21 pm

New php4 packages

Posted (January 16th, 2008 at 12:21 pm PST) by jeremy

In order to fix a couple of bugs with PHPMotion and to fix a couple of security flaws, we’re slowly rolling out new PHP4 packages to all servers. The changes are fairly minor, containing only a minor version bump and a fix to the binary installed to /usr/local/bin/php (it had previously been the CGI version rather than the command-line version). It’s already been tested on half a dozen servers without any issues so hopefully nobody (except PHPMotion users) will notice the change.


1:26 am

Billing Issues

Posted (January 15th, 2008 at 1:26 am PST) by JeffH

We are currently aware of some billing inaccuracies which we will be resolving later today. All of these issues will be resolved, and any billing errors corrected. We sincerely apologize for any hassle this has created.

– UPDATE Wed Jan 16 17:03:44 PST 2008 –

Most of you have been refunded at this time, but we have seen some cases where the refund did not go through properly. We will be looking at those as soon as we can.

There’s more information on the official blog.

– UPDATE Tue Jan 15 15:11:17 PST 2008 –

We have gone through all accounts and refunded back money for last night’s billing mistake. We have also automatically re-activated any accounts that were erroneously suspended. We are now going through the support requests remaining to check for any remaining problems. If you are still having problems, please submit a support request so we can check on it for you.

– UPDATE Tue Jan 15 04:13:17 PST 2008 –

The problem has been found and corrected. We are currently going through all erroneously charged accounts and refunding money. Stay tuned for further updates.

– UPDATE Tue Jan 15 07:25:16 PST 2008 –

Scripts are still running to correct the original problem. Last night, this guy ran a standard billing cycle to clean up stragglers from 2007. Unfortunately, the biller was run for 2008 (December 31st, 2008 to be exact). This caused everyone to be billed as if today was 2008-12-31, wreaking the havoc that we are so sorry you had to be put through.

We’re going through and fixing everything right now, but it may take several hours before all affected accounts are corrected. You will receive an email when yours is fixed, and if your credit card was charged, the charge will be refunded automatically.

We’re very very very sorry about this! There is a blog.dreamhost.com post explaining in more detail how this happened..

Also, you needn’t contact us or your credit card company/bank about this.. we’re fixing it right as you read this (and maybe already have for your account)! The best move for you is to just sit back and pretend you never got those weird billing email(s) talking about owing lots of money!

This post will be marked as resolved once (we believe) we’ve fixed everybody’s account.

Once again, we are all *extremely* sorry for the mishap. Thank you for your continued patience while we clean up our mess.


2:58 pm

Quick network upgrade

Posted (January 14th, 2008 at 2:58 pm PST) by DreamHost

We need to perform a quick network upgrade right now to alleviate a saturating port between two of our datacenters and the internet. The outage should be intermittent, and only last around 1-2 minutes. Afterwards, DreamHost should be running faster, and with less packet loss!

Edit: This turned out to be a false alarm. The network upgrades didn’t cause any noticeable outages. Other outages you may have experienced during this time were a coincidence.


1:54 pm

DreamHost Webpanel database troubles

Posted (January 14th, 2008 at 1:54 pm PST) by DreamHost

One of our central database servers is having some software trouble currently. Our webpanel as well as webmail is down as a result. We should have it back online shortly.


1:38 pm

Janky email issues

Posted (January 14th, 2008 at 1:38 pm PST) by andrea

Since earlier this morning, we’ve been experiencing slowness on the ‘janky’ cluster due to a file server problem (see your Account Status on the panel’s upper right corner to see if you’re affected). This mainly affects email access, but there are some high loads on web servers too. We’re working on resolving this as soon as possible, and appreciate your patience.

Update Mon Jan 14 15:11:45 PST 2008: The slowness with janky should be all better now.


8:29 am

Luigi being moved to new hardware

Posted (January 14th, 2008 at 8:29 am PST) by Justin@DH

The web server luigi is having hardware issues and we are moving it to new hardware. Sorry for the downtime associated with this, we are working to get it back up and working as soon as we can. Please check back for updates.


12:35 am

LAX connectivity issues

Posted (January 12th, 2008 at 12:35 am PST) by kitchen

We are currently experiencing problems with connectivity at our LAX facility. This is affecting the following clusters:

blingy
frisky
spacey
swarthy
looney

as well as dedicated server customers.

We have our network engineers looking at the problem and hope to have a resolution shortly.

Update: things appear to be back to normal. We are still looking into the issue to try to figure out what caused the network outage.


8:23 pm

Reboots on randy

Posted (January 10th, 2008 at 8:23 pm PST) by DreamHost

There was an issue with a filer that cause some servers in the randy cluster to have high loads. They are being rebooted at the moment. Sorry for the inconvenience.


5:16 pm

Emergency maintenance window

Posted (January 10th, 2008 at 5:16 pm PST) by DreamHost

Tomorrow night, January 11, 2008 at 10:00PM Pacific time, we need to perform two maintenance tasks:

1. Reboot a switch in which one of its processes if hung. It is currently passing traffic, but various Layer 3 duties it performs are in a degraded mode. We have otherwise worked around the issue, but we need to get that process un-stuck. We will be using this opportunity to upgrade the switch to the latest software version in hopes that this will not happen in the future, as this doesn’t add any time to the reload.

This should not affect general site performance, however some internal functions will be affected. If you notice sub-optimal ssh times, or other DNS related issues while logged in to our servers, they should resume in about 5 minutes when the switch comes back online.

2. Update the firmware on the disk shelves attached to a filer that crashed today. Speaking with various technicians, the current version of firmware could have been the cause of the outage in frisky today. This will cause a similar outage while the shelves reload. It should take around 5-minutes. If your data is hosted on this filer, you will not have access to it for those 5 minutes. No email or other data shall be lost, and the new firmware should make it more stable.

We’re sorry about the short notice of this window. If either of these two problems becomes worse overnight, we will go ahead and apply the fixes immediately.

Updated: 10:06 Pm Pacific: All updates went smoothly. Total downtime for affected people was <5 minutes per update. Hopefully things will be smoother here on out!


 
© 1996-2008, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
1:34 am

Sobe not so refreshing…

Posted (January 8th, 2008 at 1:34 am PST) by JeffH

Sobe appears to have some bad memory, and the admins are working on getting it failed over right now. This will not affect anyone who is not on the Sobe web server. This should be taken care of within the hour. If you are not on Sobe and are having issues, please see other status posts, and if necessary, contact support!

Update: The move hasn’t been smooth, so one of our admins is currently dedicating her full attention to Sobe. Again, this particular post does not apply to anything that is NOT Sobe. To find out if your server is Sobe, head on over to https://panel.dreamhost.com/ , and in the upper right, click “account status”. If the field, “Your Web Server”, says anything but “Sobe”, this post doesn’t apply.

This should be resolved.  Nothing to see here.  If you are still having problems, please contact support.


12:08 am

Network hiccuping

Posted (January 8th, 2008 at 12:08 am PST) by Terri

We are in the process of shuffling our network space around a bit in order to shore it up and make it more stable. However, it is causing some servers to seem slow or down. This should be resolved in an hour or less.

We apologize for the inconvenience.

Update: Ok, that didn’t go exactly as planned, but things are getting back to normal and I’ll be cleaning up any lingering issues that were caused by this little snafu. Please contact tech support with any further concerns or issues.

Further Update: Things aren’t quite back to normal, and we are working on making them so right now. We’re very sorry for the downtime, and these problems were simply not anticipated (or there would have been advance notice). We are working hard to get everything back up and running ASAP! By “seem slow or down”, we’re referring to the fact that the your server itself is actually up, running, and stable…but is not responding due to the networking problems…which may sound academic, but is a more accurate picture of what’s happening.

This is now believed to be resolved.  If you are still having problems, please contact support!


9:25 am

Happy New Year!

Posted (January 1st, 2008 at 9:25 am PST) by Josh

If you’re into that kind of thing.

We ended up getting 66 donations for a total of $5878.47 which we matched and then donated to Union Rescue Mission who in turn got it doubled for a grand total of $23,513.88!

Their site says that’s enough to feed 14,337.7317 people!

Or, in l33t-speak, LAEET TEIT people!

Great Job!


8:06 pm

System wide intermittent email sending problems

Posted (December 31st, 2007 at 8:06 pm PST) by DreamHost

We are currently experiencing system wide intermittent problems with sending email via smtp. We are working to fix, and will post more information on the status page as soon as we have it.

We believe this issue to be resolved and smtp to be working.

Happy New Year!


6:49 am

Janky File-server issues

Posted (December 31st, 2007 at 6:49 am PST) by Sandon

If your site is one one of the following servers then you pages might be loading intermittently or not at all:

adonis agamemnon aphrodite apollo aries atlas calypso charon charybdis daedalus dionysus electra erebus eros gorgon hades hecate hector helios heracles hermes hestia hypnos icarus iris jove kratos leto medusa minerva morpheus nemesis ninja odysseus oedipus pandora pegasus poseidon proteus send sisyphus smutbox synergia tartarus tethys thanatos thetis triton ysfmorpheus

We are having problems with one of our file-servers which is effecting servers in the ‘janky’ cluster and is effecting both web and mail services. Our admins are aware of the issue and working on getting it resolved ASAP.

Update 09:00 AM Pacific: It seems the motherboard in the file server went out, or something very closely related to it. The power supplies would come out of standby, try to power on the system, and it would shutoff after a few seconds. We swapped it out and are back in action! Please let us know if you are still having problems by submitting a support ticket.


3:30 pm

Last Chance to donate to Union Rescue Mission

Posted (December 30th, 2007 at 3:30 pm PST) by Josh

Happy Holidays Everybody!

Just in case you’re not already wiped out, this is just a friendly reminder that TOMORROW is the deadline for donating through our charity section to the Union Rescue Mission and have your donation not only matched by us, but again matched by a mystery benefactor.. for four times the benefit!

So far we’ve had 60 contributions for a total of $3218.47… pretty good, but we can do even better!


4:47 pm

One-Click Upgrades: WordPress 2.3.2

Posted (December 29th, 2007 at 4:47 pm PST) by Jason C

Looks like the WordPress team has decided to put out one last update before the new year. WordPress 2.3.2 contains bug fixes and a few crucial security updates, so it’s suggested you update as soon as possible via the “One-Click Installs” section of the web panel.


5:56 pm

Scheduled maintenance for patton.

Posted (December 28th, 2007 at 5:56 pm PST) by DreamHost

The clock is having some issues with skew, sometimes getting as much as 3 seconds before syncing. The battery is getting replaced, which should clear up this issue. That should happen at 9:30PST tonight. Downtime shouldn’t be more than 15 minutes.

Update: patton is back up. If you are having any issues, please contact support!


2:51 pm

blingy issues

Posted (December 27th, 2007 at 2:51 pm PST) by kitchen

The primary file server on the ‘blingy’ cluster is currently having some problems and is causing problems with all of the blingy cluster. We are working on it and should have an update shortly.

We apologize for the inconvenience.

Affected HTTP servers:
dct ermac fox ganon glassjoe himedia jax kirby link luigi mario marius neofloor2 nukem popinski scorpion sonoma spyro subzero tails toadstool yoshi zelda

Update: 6:00pm pacific: the file server appears to be back up and running normally and we are currently working to bring services back online.

Update: 6:23pm pacific: services are all back online and everything looks good. If you are still having issues, please contact support.


3:11 am

Panel downtime

Posted (December 27th, 2007 at 3:11 am PST) by DreamHost

Next Thursday morning, we will be performing some maintenance on one of the database servers which powers our Web Panel. This requires us take the Panel off line. The downtime window will begin at 2:00am PST Jan 3, and extend for two hours, until 4:00am PST Jan 3. Expected downtime within this window is 1 hour.\

Panel is back online as of 3:00am.


 
© 1996-2007, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
8:00 pm

Mysql server, Spike, has been impaled!

Posted (December 26th, 2007 at 8:00 pm PST) by Brian@dreamhost

Spike crashed, and is down for the count! We’re currently moving it to new hardware, and should have it back up soon! We apologize for the problems this causes.

The move took longer than expected, but its all back up and running. Sorry for the trouble.


4:59 am

Mysql server rintintin down

Posted (December 26th, 2007 at 4:59 am PST) by DreamHost

The database server rintintin, which serves the spacey cluster is currently down, and not responding to remote controls. An admin is heading into the data center right now to fix it.

Rintintin has been moved onto new hardware and it up and working now. I apologize for the outage.


12:17 pm

Jax needs new hardware

Posted (December 24th, 2007 at 12:17 pm PST) by Terri

Jax died and came back up with a kernel panic, so I’m moving it to new hardware right now. This should take no longer than an hour.

I’m very sorry about the inconvenience, I’ll try to get you all back up and running very quickly!

Update: It’s coming back up and seems to be running smoothly. Please contact support if you are still having problems related to the jax hardware failure.


4:40 pm

One-Click Upgrades: phpBB, phpGedView, dotProject

Posted (December 21st, 2007 at 4:40 pm PST) by Jason C

We’ve got a bit of an early Christmas (or late Chanukah) present for those of you running the above mentioned bits of web based software. In an afternoon flurry of productivity, phpBB has been upgraded to version 3.0 [Release Notes], phpGedView was bumped up to version 4.1.3 [Release Notes] and dotProject is now at version 2.1.1 [Release Notes].

If you weren’t already familiar with how to do so, you can upgrade your One-Click Installs by logging in to panel.dreamhost.com and going to the One-Click Installs page under the Goodies tab. If you happen to hit any snags in doing so, our support staff is standing by ready to help you out.

Happy upgrading, everybody!

Update: Apparently phpBB 3.0 is a pretty radical upgrade and requires a little more than our installer is configured for currently. Sorry about that! A few people have had issues with the upgrade, so we’ve gone ahead and rolled the current version on the installer back to 2.0.22. Sadly, the 3.0 upgrade will have to be pushed back until some of our more seasoned vets can figure out a proper plan of action on how to handle it safely.


12:36 pm

system problems

Posted (December 20th, 2007 at 12:36 pm PST) by danwy

We pushed out a bit of code and it’s having an adverse effect on almost all of our servers. If your site is down now, we’re aware of the issue and are working hard to get everything back up ASAP. Please keep an eye on this status post, as we’ll update it as soon as we have any further information. We’re sorry for the inconvenience this causes you.

UPDATE: The problem seems to have stemmed from an apache security upgrade that we’d tested on nearly 150 servers already over the past few weeks without issue. Some part of the upgrade procedure killed apache services at random, requiring them to be restarted. All services now appear to be back up and we apologize for the downtime.


3:32 pm

Rails upgrade addendum

Posted (December 18th, 2007 at 3:32 pm PST) by Terri

As many of you noticed, there was a slight glitch when you tried to freeze your rails. You probably got an ugly error like:

no such file to load — /home/.blah/blah/blah/config/../vendor/rails/railties/lib/initializer

Not very helpful! As it turns out, there’s a bug in rails 1.2.6 that causes this! Thankfully, it was a matter of just slightly adjusting one file, and all should be well.

Go ahead and try the freeze again, just make sure to delete the rails directory under the directory called vendor, otherwise it’ll look like it’s still broken.

I apologize if this was a pain. Again, please email tech if you’re having any further problems.


12:36 am

Dedicated Server Mail Upgrades

Posted (December 18th, 2007 at 12:36 am PST) by jordan

If you received an email last week about the mail service on your dedicated server being upgraded then you will be affected by this upgrade. If not then your upgrade will happen later this week and we will email you regarding the date and time of the upgrade. This upgrade should not cause more then 5-10 minutes of downtime where you will not be able to check email. When the upgrade is complete on all servers I will resolve this post.

UPDATE — 1:40AM PDT

The upgrades are finished. Mail service should have returned to normal and if you were upgraded you can login via user@domain.com now instead of mXXXXX mailbox names. If you are still having troubles please contact support.


4:02 pm

Rails: The Revenge!

Posted (December 17th, 2007 at 4:02 pm PST) by Terri

Just when I thought it was safe to upgrade, another version appears!

I’m going to start upgrading our servers to Rails 2.0.2. This involves packaging, testing, and then slowly rolling it out. The jump is to a 2.x version from 1.2.6, so I’m going to take my time with this one to make sure nothing goes horribly wrong, so it might be right after the new year when it’s all done, just to be safe.

For your new year’s resolution, you will freeze your rails! Login via shell, and in your rails directory, run “rake freeze_gems” – that should do the trick!

Please contact support if you have any questions or concerns.


2:19 pm

Annoying Network Issues

Posted (December 15th, 2007 at 2:19 pm PST) by Jason C

It appears as if we’re running into little flurries of bad connectivity on the network currently. Please know that we’re doing everything we can to level this back out. Hopefully it shouldn’t take too long to remedy, but we thought those of you seeing issues would appreciate the acknowledgement of a problem. As always we’ll let you know when things are taken care of.

Update: 6:52pm pacific: The network issues have been resolved. We had deployed some new networking hardware for the ‘blingy’ cluster and a couple of the ports were misconfigured, which caused random weird issues all around.


12:13 pm

Replacing hardware on Heracles

Posted (December 12th, 2007 at 12:13 pm PST) by brians

We are in the process of moving heracles to new hardware. The machine should be back online with the next 45 minutes. We sincerely apologize for any inconvenience this may have caused. Please check back for future updates.

UPDATE: The machine is back online. We will continue to actively monitor the stability. Again, we apologize for the inconvenience and appreciate your patience.


 
© 1996-2007, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
8:58 am

Control panel “status” section giving errors. (RESOLVED)

Posted (December 11th, 2007 at 8:58 am PST) by David

First let me apologize for the inconvenience. The “disk usage”, “bandwidth usage” and “mySQL usage” sections in the control panel are giving errors at the moment. Our admins are currently working on the situation and hope to have it resolved soon.

Please check back here later for updates.

UPDATED: 12/11/2007 11:49 PST, looks like there was a problem with access to the database that stores that information. The problem has been resolved and the stats are now showing up in the control panel as they should.


6:52 am

Mario server is down

Posted (December 11th, 2007 at 6:52 am PST) by Javier

last night the Mario server experienced issues that caused it to need hardware replacement, we have our admin working on getting it moved over and up and running. Sorry for the inconvenience and keep checking for an update to this issue.

UPDATE: Ok our admin was able to get the Mario server back online, it is now up and the sites on it should also be coming up fine now. If you are still having issues with your site please contact support


10:00 am

File server issue in the frisky cluster

Posted (December 9th, 2007 at 10:00 am PST) by Justin@DH

One of the file servers in the frisky cluster is having performance issues which is effecting a small number of frisky cluster web servers, causing higher loads and some apaches to respond slow. A file server admin is working on the issue now. We are working to get this issue resolved as soon as possible.

UPDATE 1:15pm (PST): We’ve corrected the problem with the file server not working correctly, but unfortunately it lost one of the drives and we’re in the process of rebuilding it. This means that there will be intermittent networking issues with the servers that mount this file server, so they’ll have higher than usual loads. This will also affect all of the mail servers in the cluster. If your server is one of the ones listed below, you may be having site or mail problems (or both). Once the drive rebuild has comepleted, these problems should subside. We apologize for the inconvenience this causes you and we’re doing everything we can to get this fully resolved.

athena barracuda belair boxster bugatti cadillac camaro carrera cayman chevelle continental corvette cutlass dart delorean edsel elcamino eldorado elise enzo esprit fairlane ferrari fiat fiero firebird galaxie george gremlin highground hornet hudson hummer hupmobile impala jaguar jetta lancer lesabre malibu manmeat mclaren merkur miata model-t montecarlo murcielago nova olds pacer packard passat phaeton pierce pinto probe prowler rambler rawstock renault riviera rolls scion sexylinker shelby skoda skylark stingray stratus studebaker tempest tesla thunderbird titan touareg towncar trans-am veyron viewgenie wrangler

UPDATE 10:30am (PST) Mon Dec 10th: Sorry for the delay in updating this post, but this problem should have been resolved sometime during the night last night. By our calculations, the drive finished rebuilding some time late last night or very early this morning. All of the loads are back down to normal and several of the tests we ran seem to be going through fine now. If you continue to have problems with this though, please don’t hesitate to contact support about it.


11:48 pm

panel.dreamhost.com issues

Posted (December 5th, 2007 at 11:48 pm PST) by kitchen

We are currently experiencing some issues with the load balancer that sits in front of panel.dreamhost.com. This is affecting other things as well, such as mailboxes.* and secure.newdream.net.

We are currently working on the issue and expect to have a resolution shortly.

We apologize for any inconvenience and thank you for your patience!

Update 12:32am pacific: things should be running smoothly again. There were some stuck processes hanging around from an earlier update on 2 of our load balanced panel machines which were making it so there was really only one machine serving the panel (and it was getting full!). We’ve killed the stuck processes (with GREAT VENGEANCE) and things are looking good now. We will continue to keep an eye on the situation, but at this point everything looks fine!

Update: 12:47am pacific: while the panel still seems to be alive and kicking, 2 of the servers are still having some issues, which may cause problems with accessing it. We’re looking into this right now to get it fixed up!

Update: 1:29am pacific: we’ve been watching it now for a little bit and it seems fine, so we’re going to resolve this once again.


2:53 pm

Promo Codes Max Discount Dropping to $50

Posted (December 4th, 2007 at 2:53 pm PST) by Josh

Effective next Tuesday, December 11th, $97 will no longer be the maximum discount a DreamHost promo code can give.. it will be dropped to $50. Keep in mind, $97 is still the total value of a referral.. so if somebody signs up with your $50 promo code you’ll still get $47 ($97-$50) credited to your account.

All existing promo codes will keep working, but will only give $50 maximum discount now. If your promo code offered extra disk, bandwidth, domain registrations, ips, etc.. those will still work, and continue to cut into the $97 you earn for a referral (so it IS possible to earn less than $47 still, if you give out feature bonuses).

That’s all really! Go forth and spread the word that this is the last week of $97 off promo codes!


4:43 pm

Sander down

Posted (December 3rd, 2007 at 4:43 pm PST) by DreamHost

One of our file servers, sander, is currently experiencing some hardware difficulties. Our filer fixing team is looking at it now and expect to have it back online shortly. More details to follow.

This was fully resolved about 15 minutes after this first status post was made. Sorry for the delayed update!


10:59 am

For people on spacey having trouble with Mac Mail

Posted (December 3rd, 2007 at 10:59 am PST) by DreamHost

When spacey moved, the IP for the email host changed to 208.113.200.50. It’s come to my attention that this has been causing problems with a lot of people receiving and sending email, especially if they had originally followed the wiki article for Mac Mail where you had to alter /etc/hosts . If you did this and you are on spacey, you either need to update /etc/hosts (with the IP above) or start using “spacey.mail.dreamhost.com” as your email server. That should fix most of the issues with this. Of course if you are still having problems after this, feel free to write into support.


12:50 pm

poseidon is sinking

Posted (December 1st, 2007 at 12:50 pm PST) by kitchen

Poseidon has had some instabilities lately, and it appears to be bad memory.

Because of this we’re going to fail it over to new hardware. This process should take around 30 minutes, and we will update here again when it is completed.

We apologize for the inconvenience and thank you for your patience!

Update: the hardware change has completed successfully and services are now all up and running. If you are experiencing any issues, please contact support!


7:57 pm

Bourbon lost a drive!

Posted (November 30th, 2007 at 7:57 pm PST) by Brian@dreamhost

Looks like bourbon’s boot drive is failing! We’re moving the server over to new hardware, so you should see about an hour of downtime.

We’re doing everything to restore functionality as quickly as possible and apologize for the problems this creates!

We’ll post an update as soon as it’s done =)

UPDATE: 08:30PM PST

Ok, everything went well, and should be back up and running. I spot checked a few sites, and they’re ok! If you’re still not seeing your domain, please contact support immediately.


9:47 am

Spacey Cluster Move! (Reminder) UPDATE

Posted (November 30th, 2007 at 9:47 am PST) by micah

This is just a reminder that the Spacey move announced a couple weeks ago will take place in approximately 12 hours from now.

— UPDATE — Sat Dec 1 06:12:59 PST 2007
Most servers are back up and running normally. A few webservers are still having issues due to a petulant file server which we’re currently rebooting (hopefully for the last time) so we’re not giving the all-clear just yet. Most services were restored within the 8 hour mark but we wanted to have everything 100% up before we resolved this post. Unfortunately it’s taking a bit longer to get these last stragglers online. Please bear with us, we’re very, very close.

— UPDATE — Sat Dec 1 06:37:54 PST 2007
All services are back online.

If you have any questions regarding the server move, please contact our support team.


 
© 1996-2007, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
1:14 pm

Rails 1.2.6

Posted (November 27th, 2007 at 1:14 pm PST) by Terri

It’s that time again! Right now I’m compiling a new version of Ruby on Rails for our servers. I will be testing and slowly rolling these out this week and next week. The upgrade should be completed by the end of next week at the latest. I will be carefully testing the upgrade to be sure it does not impact customers negatively, but I highly recommend that everyone freezes their rails – especially if you are running a Typo blog!

To freeze your rails applications, ssh into the directory that holds the application and run rake freeze_gems. If you ever need to unfreeze the application, run rake unfreeze_rails.

UPDATE: I have finished the Rails upgrade on all the servers, and it seems to have been a relatively painless process. If you have any problems regarding the upgrade, please forward your questions to tech support.


2:17 pm

Gmail Forwarding slowness (RESOLVED! 12/6)

Posted (November 17th, 2007 at 2:17 pm PST) by kitchen

Right now we are experiencing some slowdowns sending emails to Google’s GMail service. This only appears to be affecting mail being forwarded through our systems, and not sending directly through an email client using our mail servers. We believe the slowdown to be on Google’s end, however we are actively investigating the situation to see if there’s something wrong on our end causing the slowdown.

Again, email sent directly from an email client such as Thunderbird, Mac Mail, or Outlook should make it to GMail in a timely fashion, it is only mail being forwarded through our servers (forwarding-only addresses in the web panel) that is currently being affected.

Update: The mail queues are back down to normal now, and things appear to be running smoothly again!

Update: (11/28/2007 10:15 PST) Unfortunately it looks like this problem is back. Our admins are working to resolve it now. Messages are flowing, but the queue is quite backed up at the moment so it’s going to take some time.

Update: (11/29/2007 3:00 PST) We’ve moved all outgoing gmail forwards to their own server so that they’ll have dedicated processing time without affecting the rest of outgoing mail. The outgoing mail queue is still pretty full, but it’s steadily decreasing. We’re in contact with a Google support representative to get this resolved permanently. Mail to Gmail forwards are being delayed, but they are going through. A temporary workaround is to point your email forwards to a non-Gmail address or local DreamHost mailbox, as those are not affected by the delay. We’re sorry for the problems this causes you, and we’re doing all that we can to resolve this.

Update: (12/02/2007 1:00 PST) At the request of many of our customers, we are updating this page to let you know that we do not have any further information regarding this issue. We’re already in contact with Googles support team to resolve this issue, there isn’t much else we can do at this point. As far as we can tell, mail is still queued and not lost. Please try and refrain from contacting support about this for the time being, as they do not have any additional information that isn’t already posted here. As soon as we have ANY updates, we’ll be sure to post it here.

Update: (12/04/2007 9:43 PST) At the request of many of our customers, we are updating this page to let you know that we believe mail is now getting processed at a faster rate than during the weekend. The situation is not completely resolved yet as delays still exist, but there is no more problems apparent to cause further delays. We’re now waiting to catch up with the current mail flow. Again, sorry about all the inconvenience this is causing.

Update (12/05/2007 09:46 AM Pacific) Latest and greatest update! Gmail finally got back to us has whitelisted all of our outbound MX servers, plus one or two others we were using to hold the email that was being delayed by them. We cannot guarantee this will last forever, however, but for now, we’re able to send them forwarded email. We are incrementally having more of our mail servers start sending directly to gmail, instead of via our holding server. The holding server we setup has 1.4 million emails on it, and is sending them just as fast as it can/gmail will take them.

What this means to you:

1. Newly sent email to your DreamHost account that is forwarded to gmail should go through with relative speed. I am watching logs on a handful of servers and they are all sending just fine.

2. Email sent over the course of last week is sending out very quickly, and we hope will catch up today or tomorrow. 1.4 million emails is a lot of email, as you can imagine. We setup a brand new custom mail server with faster hard drives and a raid card to handle it.

We’re very sorry about how long this has taken. We have been working with gmail to get this resolved, however we are ultimately at the mercy of their servers accepting our email.

UPDATE 12/06/2007 9:07 AM Pacific:

It looks like our deferred mail server really cranked out the email yesterday, and is up to date. The only ones left in its mail queue for gmail are receiving this message back:

(host gmail-smtp-in.l.google.com[209.85.147.114] said: 450-4.2.1 The Gmail user you are trying to contact is receiving 450-4.2.1 mail at a rate that prevents additional messages from 450-4.2.1 being delivered. Please resend your message at a later 450-4.2.1 time; if the user is able to receive mail at that time, 450 4.2.1 your message will be delivered. (in reply to RCPT TO command))

Unfortunately, there is nothing we can do to prevent that. Remember, you can always have your email delivered to DreamHost for use with any IMAP/POP3 client for easy access. We have received word from them that we are whitelisted, as previously stated, and all of our inbound mail servers are successfully relaying (forwarding) their email to gmail.


10:48 am

Spacey Cluster Move!

Posted (November 16th, 2007 at 10:48 am PST) by micah

Due to space and power constraints in our primary datacenter we are moving the entire shared cluster “spacey” to our LAX facility at 10:00PM PST (GMT – 8), two weeks from today, Friday, November 30th 2007. All webservers, mail servers, file servers, and MySQL servers in the spacey cluster will be unreachable during the move, which we expect to take approximately 8 hours.

The spacey cluster contains the following web servers:

7up, amp, aquafina, arizona, barqs, bawls, boba, brisk, cactuscooler, calpico, cappuccino, caprisun, chai, clamato, codered, coffee, coke, crush, crystallight, culligan, dads, dasani, dietrite, drpepper, eggnog, endless, espresso, evian, fanta, fetish1, fiji, frappe, fresca, fuze, gatorade, geyser, gingerale, hansens, hi-c, hiball, horchata, ibc, inko, izze, jolt, koolaid, lipton, martinellis, milk, minutemaid, monster, mountaindew, mrpibb, mug, musclemilk, nehi, nesbitt, nestea, niagara, odwalla, oj, optimizer, orangebang, ovaltine, pellegrino, pepsi, perrier, plentymag, pom, powerade, propel, punch, redbull, refresco, riptide, rockstar, rootbeer, schweppes, seagull, seltzer, shasta, silk, simpledollar, slice, slim, sly, smoothie, snapple, sobe, sparkletts, sprite, squirt, stewarts, sunkist, sunnyd, tab, tampico, tang, tea, tejava, tizer, tonic, tropicana, v8, voss, welchs, wishserver, worldtown, yoohoo

If your site is on any of these servers this status post applies to you. Also, if you check the “Account Status” button on the upper right hand of the webpanel ( https://panel.dreamhost.com ) you should see your cluster under the section “Your Email Server:”. We are also sending out an email announcement about this downtime window to affected customers.

We do apologize for this inconvenience. Rest assured we will do everything possible to keep downtime to a minimum.

If you have any questions regarding the server move, please contact our support team.


7:48 am

hoover hit with UDP flood attack

Posted (November 16th, 2007 at 7:48 am PST) by Steven

We are working to block the attack and restore service as soon as possible. Please check back for further updates.

Update [8:18]: all taken care of and back to normal!

‘nother Update [10:04]: Hoover is currently experiencing very high load and is slow to respond. I’m not sure if this is related to the earlier UDP attack yet but am posting this to let you know we’re aware of its current status and are working to restore service ASAP.


10:31 am

dreamhoststatus.com maintenance

Posted (November 13th, 2007 at 10:31 am PST) by kitchen

Over the next few hours we’ll be working on the dreamhoststatus.com database and webserver, which means you might see some downtime of the site. If something critical comes up, we will cease maintenance on the website and make sure it’s back up and running for the duration.

We will try to keep it as available as possible, but it may at times become unavailable.

Thanks for your patience!

Update: We are finished with our testing, and dreamhoststatus is back to normal!


7:52 am

Bundy downtime

Posted (November 10th, 2007 at 7:52 am PST) by Steven

Unfortunately, the bundy server has been experiencing some trouble this morning. When identified an attempt was made remotely restart the server. This did not work as expected. An admin has been notified and is heading into the data center to directly restart and monitor it. I apologize for any inconvenience this has(may have) caused and thank you for your patience while this is worked on.

Update [10:53]: The server was unable to be brought back online and is now being moved to new hardware.

Update [12:10]: Apache configurations are currently being run on the server. Once this is complete the files will be copied to the server.

I apologize for the lengthy downtime regarding this. The server will be right as rain soon enough, so please hang in there!

Final Update [1:25]! The move to new hardware has been completed as has copying over of the files. All services should be restored. If you are experiencing any trouble reaching your site on bundy, please contact support@dreamhost.com so we can get you back in action as quickly as possible.


3:03 am

Updated: frisky cluster having issues

Posted (November 7th, 2007 at 3:03 am PST) by kitchen

The server ‘edsel’ is having issues mounting its file servers, which is causing home directories to be ‘missing’ and sites to be down.

We are looking into the issue and hope to have a resolution shortly.

Thank you for your patience!

Update: 3:18am pacific: we have determined the cause and have temporarily worked around the issue. Unfortunately there are still sites which are going to be down, but most things should be up and running now. We will continue to update as we progress.

Update: 3:22am pacific: we have pinpointed the cause of the issue with edsel on one of our file servers ‘fozzie’. It appears to be affecting all of the frisky cluster, and we are working to resolve this issue as fast as we can.

Update: 3:42am pacific: access to ‘fozzie’ has been restored, although communication with it is still slow. We are continuing our investigation and hope to have a resolution soon!

Update: 3:49am pacific: guess the slowness was just everything getting all nice and cached everywhere. The Fozz appears to be back!

If you are still having problems, please contact support!


3:28 pm

cerritos isn’t very social

Posted (November 1st, 2007 at 3:28 pm PST) by kitchen

Cerritos appears to be having some issues with one of its network interfaces, so we’re going to give it new hardware.

This move should take about 30-45 minutes and we will post again here with updates as things progress.

Hopefully this will alleviate some of the issues it’s been having!

Update 3:54PM pacific: the hardware failover has been completed and services are being configured. We’ll update again when everything is completed!
Update 4:14PM pacific: the failover process has completed, please contact support if you are having any issues!


8:56 pm

5.6 Earthquake in San Francisco

Posted (October 30th, 2007 at 8:56 pm PST) by kitchen

Just writing to let everyone know that this doesn’t appear to have affected our San Francisco servers at all, nor our LA datacenters (didn’t feel it at all in LA)

Hope everything is ok up in San Francisco! We’ll be thinking about you!

For those interested, here’s information from the USGS about the earthquake…

http://quake.wr.usgs.gov/recenteqs/Maps/122-37.html


6:46 pm

File server Bambi having problems

Posted (October 27th, 2007 at 6:46 pm PST) by Brian@dreamhost

The file server Bambi is currently down and has been having issues for the last hour. We’re working on restoring service to it as fast as possible. This can effect the following machines:

7up amp aquafina arizona barqs bawls boba brisk cactuscooler calpico cappuccino caprisun chai clamato codered coffee coke crush crystallight culligan dads dasani dietrite drpepper eggnog endless espresso evian fanta fetish1 fiji frappe fresca fuze gatorade geyser gingerale hansens hi-c hiball horchata ibc inko izze jolt koolaid lipton martinellis milk minutemaid monster mountaindew mrpibb mug musclemilk nehi nesbitt nestea niagara odwalla oj optimizer orangebang ovaltine pellegrino pepsi perrier plentymag pom powerade propel punch redbull refresco riptide rockstar rootbeer schweppes seagull seltzer shasta silk simpledollar slice slim sly smoothie snapple sobe sparkletts sprite squirt stewarts sunkist sunnyd tab tampico tang tea tejava tizer tonic tropicana v8 voss welchs wishserver worldtown yoohoo

We apologize for the problems this cause, and should have it fixed soon!

Update: 8:30pm pacific: We’ve taken care of the problem and the file server is back up and running. Services should all be up and running fine now, please contact support if you are still having problems!


 
© 1996-2007, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
5:04 pm

One-Click Installs: WordPress 2.3.1

Posted (October 27th, 2007 at 5:04 pm PST) by Jason C

The version of WordPress on the One-Click Installer panel has been updated to 2.3.1. This is both an update for security and bug fixes, so it is suggested you update your installs as soon as possible. For more information on what all got changed, check out the official release post over on the WordPress blog.


3:14 pm

Moving triton to new hardware

Posted (October 27th, 2007 at 3:14 pm PST) by DreamHost

The webserver triton has experienced some problems and is being moved onto new hardware. I’ll post an update when finished.

Update; 3:54pm: I’ve just finished the move and triton is now up and working. If you have sites hosted on triton and they are having any problems please contact support.


10:09 am

Network Outage

Posted (October 26th, 2007 at 10:09 am PST) by brians

We are experiencing network problems at the moment. This will account for for downtime, delay, etc. that you are encountering when accessing your site, email, etc. Our Admins are currently working on the issue as I type. We sincerely apologize for any inconvenience this has caused. Please check back for future updates.

UPDATE 10:39 PST

The network problems have been resolved. A DDOS attack launched on a customers site was the root cause of recent downtime. Again, we apologize the inconvenience this has caused and truly appreciate your continued patience and loyalty.


9:53 pm

Major Outage

Posted (October 25th, 2007 at 9:53 pm PST) by DreamHost

We are currently experiencing what seems to be a system wide outage. We’ll post more information here as soon as we have it.

Update: Things seem to be working again, we apologize for the outage.


9:08 pm

File Server Maintenance

Posted (October 24th, 2007 at 9:08 pm PST) by Patrick

One of our file servers for the spacey cluster has a dead battery! It will be going down around 10pm PDT for a replacement. Downtime should be no longer than 5 minutes.


12:32 pm

Scheduled maintenance on central services

Posted (October 24th, 2007 at 12:32 pm PST) by jordan

Thursday October 25th at 9:00PM PDT we will be performing maintenance on the following services.

Dreamhost Panel (panel.dreamhost.com)
Dreamhost.com
Mailboxes Web interface (http://mailboxes.dreamhost.com)
Dreamhost Scripts (http://scripts.dreamhost.com)

We do not expect any downtime of these services but there may be a few short outages while we shuffle things around. The purpose of this maintenance is to improve the performance and reliability of all of these services by moving them to bigger and better load balancers.

Update; 10:30pm: We’ve found some problems with database creation and administration on the Webpanel, (and one-click installs). The admin team is working to fix it right now, so there’s no need to contact us about it. We’ll post an update when it’s resolved.

– Update Thu Oct 25 23:33:33 PDT 2007 –

Database creation and administration via the web panel is working properly again. The sign-up page on dreamhost.com was broken for awhile. We’ve had to make some last minute changes to get account registration working again. If you have any problems accessing the sign-up page, you can reach it directly here:

https://signup.dreamhost.com

If you do encounter any broken links to the sign-up page, please let support know.


12:28 am

Server, LeSabre, Being Failed Over

Posted (October 24th, 2007 at 12:28 am PST) by JeffH

The web server, LeSabre is currently having hardware issues of an as-of-yet undetermined nature. We will be failing it over to new hardware and posting updates as available. We apologize for the inconvenience. This will only affect users on the server, “LeSabre”. Email service will not be affected…

-Update- 3:38am Pacific

The failover process is complete! Everything should be running now on the new hardware. If you are still having issues, please contact support!


9:31 pm

Secondary name server experiencing packet loss

Posted (October 23rd, 2007 at 9:31 pm PST) by Brian@dreamhost

The colocation facility that houses our secondary name server, ns2.dreamhost.com is currently experiencing some network problems, causing some servers to appear down, or unavailable.

We’re currently working with our provider to fix this issue, and will post more information as soon as we’re updated by by their command center.

–UPDATE– Tue Oct 23 22:59:36 PDT 2007

While our secondary NS colocation facility insists the problem is on their end, we continue to witness issues across the board. This is causing many sites to be inaccessible though the majority are up and running fine. We are looking further into the issue and raising this post to medium severity. We sincerely apologize for any downtime this causes. We don’t like it any more than you do.

– UPDATE Wed Oct 24 05:43:24 PDT 2007 –

All is clear on the westward front! The packet loss to our off site facility has actually been resolved for a few hours. For the sake of being prudent however, we refrained from resolving the issue until everything cleared up. ns2.dreamhost.com is back up and fully functional. If you are experiencing any problems whatsoever, please contact support.


11:14 pm

pierce getting new hardware

Posted (October 21st, 2007 at 11:14 pm PST) by kitchen

The server ‘pierce’ appears to have a malfunctioning network card, so we’re going to move it over to new hardware.

This process takes about 30-45 minutes, and we will post an update here when it is completed.

Update: 11:37pm pacific: The hardware change has been completed and services are coming up. Everything should be fully completed in a few more minutes!

Update: 11:43pm pacific: The failover process is complete! Everything should be running now on the new hardware. If you are still having issues, please contact support!

Sorry for any inconvenience!


10:17 pm

Core Router Maintenance

Posted (October 19th, 2007 at 10:17 pm PST) by kitchen

As planned, we are beginning our emergency core router maintenance right now.

Our planned downtime window is 2 hours, during which all services including email, webmail, websites, ftp, ssh and the web panel will be offline. We will post updates here as things progress, so please keep an eye out and we’ll let you know when things are back online!

We are going to try to take care of as many things as we can during this maintenance window, so to minimize required outages in the future.

We greatly apologize for the inconvenience and thank all of you for your patience!

— UPDATE 12:18 AM Sat Oct 20

The router is back up, again, after the redundancy test we performed on the upgrade failed miserably. We are now running the new version of IOS, fortunately, so all is not lost. We are currently seeing if the network and the servers are back up and running.

— UPDATE 1:34 AM Sat Oct 20

We believe that all systems are Go! Please write in to support if you are still having any problems. We will continue to monitor this throughout the night. We apologize for the extended outage, we will be contacting Cisco regarding that bug in the redundant failover system.


 
© 1996-2007, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
9:02 pm

Troubles with Janky

Posted (October 19th, 2007 at 9:02 pm PST) by DreamHost

We’re having some performance problems with one of the file servers in the Janky cluster. You can check your cluster by clicking the Account Status link at the upper right of the Webpanel, it’s the Email Server. We’ve got an admin headed in right now to work on it and hopefully get it fixed asap.


1:48 pm

File Server getting a new head

Posted (October 19th, 2007 at 1:48 pm PST) by Terri

One of our file servers has faulty hardware, so our admins will be swapping out the head unit. They will, of course, do this as quickly as possible, but this will create downtime for customers on the following servers:

adonis aphrodite apollo aries baachus barefoottours charon charybdis chronos daedalus dione dionysus erebus erida eros ganesha gorgon hades helios hephaestus hera heracles hermes hestia hypnos icarus iris jove kratos medusa mf1 mf2 minerva morpheus nemesis odysseus pan pegasus persephone poseidon rhea sisyphus tartarus tethys thanatos thetis triton zeus

This will also affect email for people on these servers.

We will post more information as it becomes available. We apologize for the inconvenience.

Update: The file server’s hardware was replaced superfast, and we’re finishing up cleaning up any remaining issues. However, if you are still having a problem, please email tech support.


1:51 am

sanjay / lovejoy filer issues

Posted (October 19th, 2007 at 1:51 am PST) by kitchen

One of the file servers for the sanjay and lovejoy machines experienced a filesystem failure this evening.

We are currently running a repair on it, which is taking some time, and will take some time more before it finishes.

We will post an update here later when it has completed and things are back online.

We are very sorry about the very late posting on this, there was a miscommunication earlier and it never got posted.


9:35 pm

euro getting new hardware

Posted (October 18th, 2007 at 9:35 pm PST) by kitchen

The machine ‘euro’ appears to have a faulty hard drive which is causing some problems this evening. Because of this we are going to move it to new hardware.

This process should take about 30 minutes, and we will post an update here when it is completed.

Update: 10:13pm pacific: the hardware switch is complete and services should be coming up. We will post another update when everything is ready to go!

Update: 10:42pm pacific: the failover process has been completed and services should all be running! If you have any problems, please contact support!

Thank you for your patience!


2:03 pm

Major networking issues

Posted (October 18th, 2007 at 2:03 pm PST) by kitchen

We are currently experiencing some major issues with one of our core routers which is causing enormous amounts of packet loss to and from our network.

We have our network engineering team working on the issue right now, and things are definitely better but we are still experiencing some issues. We hope to have a resolution very shortly.

We very sincerely apologize for the inconvenience, we understand how frustrating this is for everyone.

– UPDATE Thu Oct 18 15:38:27 PDT 2007 –

Due to the issues we are currently experiencing, we are moving some of tomorrow night’s scheduled maintenance up. There will still be a maintenance window tomorrow night, but the time required will be much less. Currently services are being restored. We will post another update as soon as we are back in the clear.

– UPDATE Thu Oct 18 16:50 Pacific 2007 –

We believe that all major issues are currently resolved. We are still doing minor cleanup work, please write in to technical support if you are still having problems.

The catalyst to this outage was a scheduled turn-up of a new internet uplink, another gigabit of bandwidth. This is normally a smooth transition, our routers add the new link to the routing tables, and we slowly migrate bandwidth usage on to the new path until the uplinks are balanced. Yesterday during the testing phase, the link went live unexpectedly, since the link was supposed to be down on the remote side, causing our secondary router to run out of memory. The link was nearly immediately shut down and the router recovered. We then knew we should put it on to our primary router.

Today we did the scheduled turn up with our bandwidth provider on the phone. All went smoothly for about 30 minutes. The router brought the link online and began using it immediately. 30 minutes later, our secondary router promptly crashed, running out of memory again for no apparent reason. It took us around 20 minutes to diagnose that, as network and router access was extremely slow. Once we determined that, we immediately got our network consultant on the phone who recommended and immediate shutdown of our secondary router, as these were the issues we saw a few months ago and thought we had fixed.

We moved the two live uplinks from the secondary router to the primary one and brought them online. Once the secondary router was cut off from the primary one, the failover software performed correctly and migrated the default gateways over, and the network started to look up. Due to a problem in the version of IOS we are running on our router, we have been spending the last hour or so re-balancing the load on our uplinks. This bug should be fixed in tomorrow nights software upgrade.

Most of the servers recovered on their own, with only around 30 or so needing reboots. We are still cleaning up some loose ends. We did not do the software upgrade today while the network was broken due to the inherent risk involved. Tomorrow night our network consultant will be on-site to ensure the upgrade goes smoothly. Tomorrow during the day we will be rebuilding our secondary router to have the same hardware as our primary one. We have all of the hardware on the shelf, it was just waiting for tomorrow nights window to get the router offloaded.


2:41 pm

Core router upgrades

Posted (October 17th, 2007 at 2:41 pm PST) by DreamHost

This Friday, October 19th, 2007 at 10:00PM Pacific time we will be performing emergency maintenance on our core routers. We are going to be upgrading the operating system on one of our core routers in our main facility, and one primary distribution router at our LAX facility. The total downtime shouldn’t be much longer than it takes these to reboot into the newer operating system, and reestablish the links to the outside world. The maintenance window is two hours, though we hope everything from a network standpoint will be completed in 30 minutes.

These upgrades will address many of the issues we have been having over the past months. Cisco has recommended this version of IOS to potentially solve a network flooding issue we have seen intermittently, and if it does not, will allow us to escalate the case even further. The upgrade will also give us support for new hardware we have purchased for the routers themselves, increased diagnostic gathering capability, and bring online our redundant supervisor line card.

We apologize for the short notice and overall inconvenience of this upgrade, however this will hopefully fix various outstanding bugs that crop up during the day. We are attempting to do as much as possible with this reboot to minimize further downtime.

All systems will be affected by this upgrade.


1:21 pm

Policy Clarification: Personal storage & back-ups…

Posted (October 17th, 2007 at 1:21 pm PST) by DreamHost

This is a clarification regarding existing DreamHost policies on the use of services for personal storage/back-up purposes. No service policy has been changed.

Please note that DreamHost services are intended for the purpose of hosting web sites (item #6, Material Products), and files uploaded to your accounts should be provided with the intent of distributing them to others from your hosted sites. With exception of our Files Forever service, we do not support the use of our hosting services for personal file storage or back-up purposes and the use of your hosting account for that purpose is prohibited.

For those concerned about data integrity of site content, we do maintain back-ups of all customer site and email data through our snapshot back-up system. As an added precaution, we also recommend maintaining localized back-ups of any especially important files on your own computer.

If you have any questions about this or any of our other policies, please do not hesitate to contact technical support for further clarification.


12:08 pm

Network Issue

Posted (October 17th, 2007 at 12:08 pm PST) by Terri

One of our core routers is dropping about half of its packets at the moment. Our admin staff is working very hard to resolve this problem right now. This will cause outages and slowdowns for the majority of our customers.

We apologize for the inconvenience this is causing. We will post again when we have more information.

Update: The router ran out of memory and caused it to drop packets. It is slowly recovering right now, so services may remain slow for a bit, but will be improving very soon. We’re still digging into the router to see why it ran out of memory.

Update: This is resolved now. All issues related to the network should be resolved. If you are still encountering a problem, please contact us via tech support. Our efforts to optimize our network continue.


1:52 am

Scheduled downtime of central services.

Posted (October 17th, 2007 at 1:52 am PST) by jordan

Tomorrow Oct 18th 2007 at around 12:30-1AM PST we will be performing maintenance on some of our central services. This maintenance will include..

discussion.dreamhost.com (Dreamhost discussion forums)
dreambook.com (Dreambook)
wiki.dreamhost.com (Customer Wiki)
Webftp
Cloaked domains (Any domains that you have set to use cloaking in the panel)

We expect the outage to not last longer then an hour. The purpose of this upgrade is to move these services onto redundant new load balancers which will improve their performance.

UPDATE — 2:16:37 PDT OCT 18 2007

The upgrades have been completed, most services affected have returned to normal with an exception to discussion.dreamhost.com and wiki.dreamhost.com. The ip of these had to be changed so dns must propagate again for them. This should be completed in a couple hours for most people but may take longer for others. If you notice any problems please contact support with information regarding the outage.


9:53 am

Janky mail server issues

Posted (October 16th, 2007 at 9:53 am PST) by andrea

Very sorry about the recurring issues affecting mail service on our ‘janky’ server cluster. You may be experiencing slowness or unavailable mail if your mail server is listed as janky under Account Status on the control panel (upper right corner).

We apologize for all the inconvenienence this has been causing you, and we’re working on resolving this for good. Please check back for updates.

–Update– Tue Oct 16 11:08:40 PDT 2007

As of now the symptoms seem to have subsided for the moment. One of the file servers for janky is having problems keeping up with demand. We’re migrating a good portion of its data to another filer server right now. These measures should resolve the issue for good. We apologize for the recurrence of the problems, we’re confident they’ll be better shortly.


 
© 1996-2007, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
1:24 am

Problem with spacey cluster.

Posted (October 16th, 2007 at 1:24 am PST) by jordan

We are currently having problems with the spacey clusters. The problem is being caused by a filer that is having NFS problems. It is being taken down for a reboot at this time and should be back up shortly. We apologize for the inconvenience and will update this post as soon as the filer comes back up.

UPDATE–
The problem has been fixed, after a phone call to a vendor we have found the problem with the filer and corrected it.


1:33 am

File server ‘refrigerator’ blew a drive.

Posted (October 15th, 2007 at 1:33 am PST) by kitchen

The file server ‘refrigerator’, one of the file servers for the frisky cluster has blown a drive and is rebuilding its array. Because of this the system is responding a bit more slowly than usual, and will probably cause intermittent slowness of shell access, websites, email and webmail for users whose homedirs are on refrigerator.

As of right now, the rebuild is 5% complete, and will probably take another few hours to complete.

We want to emphasize that no data has been lost, and no data is in danger of being lost, drive failures on file servers are fairly common when it has over 100 drives.

Update: 9:51am pacific: the array rebuild process has completed. If you are having issues, please contact support!


8:55 pm

Moving hades onto new hardware

Posted (October 13th, 2007 at 8:55 pm PST) by DreamHost

The webserver hades has experienced a disk drive failure and is being moved onto new hardware. No customer data was lost, and the new hades should be online within a half hour.

Update 9:50pm: hades should be all up and running again, please contact support if you have any sites hosted on hades that aren’t working.


11:37 am

Site slowdowns for some customers

Posted (October 12th, 2007 at 11:37 am PST) by jeremy

Another filer issue this morning caused some high loads and site slowness for the following servers. It’s now been fixed and we’re working on resolving the issue permanently.

7up amp aquafina arizona barqs bawls boba brisk cactuscooler calpico cappuccino caprisun chai clamato codered coffee coke crush crystallight culligan dads dasani dietrite drpepper eggnog endless espresso evian fanta fetish1 fiji frappe fresca fuze gatorade geyser gingerale hansens hi-c hiball horchata ibc inko izze jolt koolaid lipton martinellis milk minutemaid monster mountaindew mrpibb mug musclemilk nehi nesbitt nestea niagara odwalla oj optimizer orangebang ovaltine pellegrino pepsi perrier plentymag pom powerade propel punch redbull refresco riptide rockstar rootbeer schweppes seagull seltzer shasta silk simpledollar slice slim sly smoothie snapple sobe sparkletts sprite squirt stewarts sunkist sunnyd tab tampico tang tea tejava tizer tonic tropicana v8 voss welchs wishserver worldtown yoohoo


3:12 pm

missing email addresses

Posted (October 11th, 2007 at 3:12 pm PST) by danwy

We’ve received multiple support requests from customers who are saying that their email addresses are missing from the panel and are receiving ‘unknown user’ errors. We are aware of this situation and are working on restoring everything. If this is happening to you, please contact support right away. We apologize for any inconvenience this causes you.

Update: The cause of the problem has been discovered and we are in the middle of restoring all missing email addresses.

ETA: 1 to 2 hours.

Update: All data has been restored. If you are still missing any email addresses on your Manage Addresses control panel page, please contact support for further assistance and we will get you taken care of as quickly as possible!

Sorry for the inconvenience!


9:09 am

Janky mail problems

Posted (October 10th, 2007 at 9:09 am PST) by andrea

We’re currently experiencing very high loads on janky mail servers. This is causing your mail to be slow or inaccessible. You may be affected if your mail server is listed as ‘Janky’ under Account Status in the control panel (upper right corner).

Very sorry about the inconvenience, we’re doing all we can to alleviate the situation as soon as possible. Please check back for updates.

Update @ 10am PST:
The problem turned out to be file server related, so the admins offloaded the machine to share the load with another file server. We’ve been monitoring the load for the past hour or so, and it’s now stable. If you’re still having mail issues it’s most likely unrelated, so please submit a new requests with as many details as possible, so we can assist you asap. Sorry again about the inconvenience this have caused you.


7:03 am

Server moves!

Posted (October 9th, 2007 at 7:03 am PST) by DreamHost

Wednesday Morning at Midnight, October 10, 2007, we will be moving some servers in our datacenter in an effort to better make use of our rack space and available power. The estimated downtime is 5 minutes for these three servers:

vito, baseline, fin

We are also moving a handful of internal use servers as well, but that downtime will be transparent to you!

Update! 12:48AM Pacific: All finished! Everything should be back up and running again. Please contact support if you are having any problems.


6:53 am

Server martinellis being moved to new hardware

Posted (October 5th, 2007 at 6:53 am PST) by chih

Martinellis just had a hardware failure, and we’re moving the server over to new hardware right now. It should take an hour to get everything up and running again. Sorry about this!

Update: Martinellis is back up and running. Please report any lingering issues to tech support. Thank you for your patience!


6:23 am

File Server Issue

Posted (October 4th, 2007 at 6:23 am PST) by Terri

A file server is having an unknown issue at this time. We are working to resolve this issue, and will update as soon as we know more.

Servers that are possibly affected are:

7up amp aquafina arizona barqs bawls boba brisk cactuscooler calpico cappuccino caprisun chai clamato codered coffee coke crush crystallight culligan dads dasani dietrite drpepper eggnog endless espresso evian fanta fetish1 fiji frappe fresca fuze gatorade geyser gingerale hansens hi-c hiball horchata ibc inko izze jolt koolaid lipton martinellis milk minutemaid monster mountaindew mrpibb mug musclemilk nehi nesbitt nestea niagara odwalla oj optimizer orangebang ovaltine pellegrino pepsi perrier pom powerade propel punch redbull refresco riptide rockstar rootbeer schweppes seagull seltzer shasta silk simpledollar slice slim sly smoothie snapple sobe sparkletts sprite squirt stewarts sunkist sunnyd tab tampico tang tea tejava tizer tonic tropicana v8 voss welchs wishserver worldtown yoohoo

We apologize for the inconvenience.

Update: We have found the cause of the outage and have resolved it. There appears to be an inefficiency with this type of file server regarding how it treats backups. We will be investigating this issue further and will be preventing it from happening again.


2:31 pm

Major networking issue.

Posted (October 2nd, 2007 at 2:31 pm PST) by DreamHost

We are currently experiencing a major issue with our network that is affecting all sites and services. The admin team is working to fix things right now. We’ll post when we have more information.

Update: Services seem to be returning to normal, watch this space for more updates.


 
© 1996-2007, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
9:40 am

Last Chance to Donate

Posted (October 1st, 2007 at 9:40 am PST) by Josh

To Public Counsel and Apraxia-KIDS!

Go to our Home > Charity area to give.. and remember all donations are matched by us!

So far we’ve only gotten 11 donations for $387.86 to Apraxia-KIDS and 13 donations for $416.02 to Public Counsel!


8:28 pm

Moving atlantic onto new hardware

Posted (September 29th, 2007 at 8:28 pm PST) by DreamHost

The webserver atlantic has been suffering from poor stability lately, so I’m going to be moving it onto new hardware. This process should take less than thirty minutes, and I’ll post when I’ve finished.

Update 9:37pm. Atlantic is now on new hardware and appears to be running fine. If you have a site hosted on atlantic and it’s having any trouble please contact support.


1:31 pm

geyser not spouting

Posted (September 28th, 2007 at 1:31 pm PST) by Terri

Geyser has experienced a failure. It is not booting, so we have to switch everyone over to new hardware. This will take no longer than an hour to get everyone back up and running.

We apologize for the inconvenience.

Geyser is back up and running and I even found the source of the crash, which has been fixed. Please inform tech support of any lingering issues.


4:18 pm

One-Click Installs: WordPress updated to 2.3

Posted (September 27th, 2007 at 4:18 pm PST) by DreamHost

WordPress 2.3 adds quite a few new features, which you can read more about on the release page:

WordPress 2.3 Release Info

A few important notes before you upgrade:

-Please check your plugins and themes for compatibility before the upgrade. You can do so at the links below:
WordPress 2.3 Plugin Compatibility List
WordPress 2.3 Theme Compatibility List

-The new version of WordPress has an automatic functionality to send information to WordPress servers concerning your software/plugin versions. Since some users have objected to the idea of this data being sent, I’ve automatically included two plugins: one that allows you disable WordPress update notification, and one that allows you to disable plugin update notification. More information about this issue can be viewed in this post by one of the lead developers of WordPress.

As always, WordPress may be updated by logging in to panel.dreamhost.com, clicking on the Goodies tab on the left side, and then clicking on the One-Click Installs page.


9:50 am

Janky file server slowness

Posted (September 27th, 2007 at 9:50 am PST) by Justin@DH

We’re currently experiencing problems with one file server that some home directories are mounted under. We’ve got the system administrators working on the problem now and the speeds will be improving throughout the day as work is done on the file server. Some effects caused by the file server are slow webpage loading, slow email and webmail. We are extremely sorry for the inconvenience that this has caused to those of you on the janky cluster. Please check back for updates.


4:28 am

Server titan getting new hardware

Posted (September 27th, 2007 at 4:28 am PST) by kitchen

The server titan has suffered a boot drive failure, and will not boot. We are moving it over to new hardware right now, a process which will take approximately 30 minutes.

We will post an update here when it is finished.

Update: 4:55am pacific: the hardware change has been completed. services will be coming up over the next few minutes.

Please write to support if you are still experiencing issues!

Thank you for your patience!


8:50 am

Network Problems

Posted (September 26th, 2007 at 8:50 am PST) by brians

We are experiencing intermittent network problems. This will explain any lag / downtime that you may encounter while attempting to access your email, website, etc. Our Admins are diligently working towards resolving issue. We sincerely apologize for the inconvenience this has caused and truly appreciate your continued patience. Please check back here for future updates.


10:03 pm

More graveyard server moves!

Posted (September 25th, 2007 at 10:03 pm PST) by kitchen

The following mysql servers will be moved starting at 1am pacific this evening.

bast
heathcliff
juno
guinness
supercat

Each machine will be down for about 15 minutes. We will update this post when the moves are started, and again when they are completed.

Update: 1:25am: The moves are now starting. bast and heathcliff are first on the list.
Update: 1:42am: heathcliff and bast are now back up and running. Starting on juno, guinness, and supercat.
Update: 2:00am: juno, guiness, and supercat are all back up and running.

Thank you for your patience!

Sincerely,
The Happy DreamHost Late Late Late Late Late Show Team.


5:58 pm

Graveyard server moves

Posted (September 24th, 2007 at 5:58 pm PST) by kitchen

Continuing with our datacenter re-arranging tonight we’ll be moving 2 mysqls from one rack to another.

mimsey
figaro

We will be moving these at 1am pacific time, and the downtime should be no more than 15 minutes. We will post another update here when the moves are complete.

Update: 1:06am pacific: The moves are beginning now. We will post another update when they are completed.
Update: 1:19am pacific: The moves have been completed and everything should be fine!

Thank you for your patience!

Sincerely,
The Happy DreamHost Graveyard Shift Team.


3:43 pm

File server chiller died *resolved*

Posted (September 24th, 2007 at 3:43 pm PST) by Terri

The head unit of one of our file servers, chiller, has failed. Admins are working very hard right now to replace it.

This will affect people on servers:

andor
atomic
cyrax
dali
dimarco
frigga
fud
genki
gradius
hug
kenobi
lilac
limbo-swarthy
narcissus
noid
postac
rock
samus
sektor
slash
slaw
soda
squirm
strange
thorin
tron
van
vidar
vlad
wiggle
zebes

Additionally, customers who are on these servers may experience issues with mail.

We apologize for the inconvenience, and we will provide more information on this as it becomes available.

UPDATE: The file server has been repaired and up and running. Everything should be running smoothly. If you still have concerns about your server, please email tech support.


 
© 1996-2007, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
5:07 pm

dreamhoststatus.com maintanence

Posted (September 22nd, 2007 at 5:07 pm PST) by kitchen

We will be doing some work with dreamhoststatus.com for the next few hours, so access to it may be sporadic.

You may get strange errors and/or broken pages while we’re doing this, but hopefully it won’t take very long :)

We apologize for any inconvenience this may cause!

Sincerely,
The Happy DreamHost Break The Status Page Robot

Update: 9/24 2:24am pacific: Ooops, forgot to resolve this. It was only unavailable for a few minutes, and is back up and running fine now!


8:46 am

Krakatoa mysql server problems

Posted (September 20th, 2007 at 8:46 am PST) by Justin@DH

The mysql server krakatoa is experiencing some hardware problems. Our admins are checking out the issue now. As soon as we have further info this post will be updated. Our apologies for the downtime.

–UPDATE– Thu Sep 20 09:54:35 PDT 2007

Krakatoa has been successfully booted and we’re copying data off to a new server right now. It seems a RAID issue with one of the partitions was making it inaccessible. We’ll resolve this post once MySQL is up and running on the new server. We appreciate your patience.

–UPDATE– Thu Sep 20 11:43:02 PDT 2007

All of Krakatoa’s MySQL databases have been moved over to server Sifl. They’re all up and running normally again. Please contact support if you are still having problems with your databases.


11:04 am

Email Address Problems

Posted (September 19th, 2007 at 11:04 am PST) by danwy

Some of our customers have been reporting that they’re having issues with editing email addresses, or that all of their addresses are missing via the web panel. We’ve narrowed it down to some MySQL tables that contained address information which were being overwritten by an automated processes. We’re working on getting it back up and running as soon as possible and will restore all of the address information that we have from backups at that time. We’ll make an update here once all the data is restored. Sorry for any inconvenience this causes you.

UPDATE! This should have been fixed up last night (the 19th), sorry for the lack of updates. If you’re still having problems with any of your email addresses not showing up correctly in the panel, please contact support.


8:30 pm

File server shuffling

Posted (September 18th, 2007 at 8:30 pm PST) by DreamHost

On Thursday night, September 20th starting at 10pm Pacific time, we will be shuffling a handful of fileservers for the Randy and Spunky clusters. Each move should take no longer than 15-20 minutes per file server. This is to take inefficiency out of our racks and allow us to deploy even more newer/better/faster file servers! During the time that the file servers are down, your websites and email will be inaccessible, however, no email will be lost. We hope to finish by midnight.

These are the file servers which we are moving. If you have files on these servers they will be inaccessible for about 15-20 minutes.

compressor curler smasher tanker churner digger potter scraper charger nailer screwer switcher

Update @ 10:53 PM, September 20, 2007: We are finished, and have been for half an hour more or less. We wound up only needed to move two of the filers listed above, and they were the two smallest of the batch. No servers crashed, and total downtime was around 4 minutes each filer. Please contact technical support if you are having any problems!


6:38 pm

Email configuration restored

Posted (September 18th, 2007 at 6:38 pm PST) by pete

Some customers have been experiencing a problem where your Manage Addresses control panel showed few or no entries. This has been fixed, and will be monitored for future problems. However, because a database restoration had to be made, any changes you have made to your email since roughly 10pm pacific time last night (09/17/2007) have been lost and must be redone. If you experience problems with your email panel, or with sending or receiving email, please contact our technical support department with as detailed a description as you can provide.

We apologize for the inconvenience.

** Update

It turns out the backup was restored from 09/13, not 09/17 as previously thought. A data merge has been performed, and those of you who added email accounts since 9/13 but now noticed them missing should be fixed. If you deleted an email account since 9/13, it may show up in the panel again. Simply delete the address again to resolve this issue.

Once again, we apologize for the trouble!


5:22 pm

One-Click Installs: MediaWiki updated to 1.11.0, PhpGedView updated to 4.1.1

Posted (September 17th, 2007 at 5:22 pm PST) by DreamHost

MediaWiki 1.11.0 has been released with some fairly serious security upgrades:
MediaWiki 1.11.0 release notes

PhpGedView 4.1.1 comes with numerous bugfixes but no new features:
PhpGedView Homepage

As always, you can upgrade your One-Click Installs by logging in to panel.dreamhost.com and going to the One-Click Installs page under the Goodies tab. Also, take care to read the instructions in the post-upgrade e-mail for MediaWiki, as they contain important instructions are almost always necessary to follow for your newly upgraded MediaWiki run properly!


2:49 pm

Network Issues causing downtime and slowness

Posted (September 17th, 2007 at 2:49 pm PST) by Ralph

We are currently seeing some network issues that are causing downtime and slowness to one cage in one of our datacenters. We are actively investigating the issue now and will update this post as soon as we have further information. Sorry for the inconvenience this is causing you, we will have everything working again as soon as we can. Please keep your eye on this post for further information.

UPDATE: Network issues have ceased and we are seeing things returning back to normal. We also noticed that there was DoS attack going on as well which we have handled. All traffic is now moving normally. We also believe that some additions we made in the data center contributed to the issues as well. Sorry about this. Things should be much improved from when this was first posted.


2:26 pm

Server daedalus getting new hardware

Posted (September 17th, 2007 at 2:26 pm PST) by kitchen

The server ‘daedalus’ had a hardware failure, so we’re moving it over to new hardware. This process should take about 30 minutes, and we will update this post when it is completed.

We apologize for any inconvenience.

Update: 3:03PM pacific: We’ve successfully switch it over to the new hardware. Services should be coming up steadily. We will post another update again when everything is complete.

Update: 3:13pm pacific: The hardware move process has been completed. If you have any issues with daedalus, please contact support!


9:42 am

resolved! ikari web server offline

Posted (September 14th, 2007 at 9:42 am PST) by Steven

Well, not entirely. The server is up and running but is not serving web sites. Our admins are already looking into the problem and will have it resolved as soon as possible. Please check back here for further updates.

Update: The server is up and running now. If your site is hosted on this server and it’s still not accessible please contact support@dreamhost.com


12:27 am

Server Relocation Bonanza Redux!

Posted (September 12th, 2007 at 12:27 am PST) by Patrick

What, you thought we really meant we were almost done back in January? Silly you!

The following servers will be moved Wednesday night beginning at 11pm PDT.

absinthe, admiralty, aero, bubbleyum, bundy, butler, cahuenga, caramello, centinela, chex, cordial, dorito, fairfax, fireballs, garvey, gobstopper, grappa, holt, jefferson, lankershim, larchmont, lemonhead, licorice, lucas, malt, moorpark, nash, necco, nerds, paramount, payday, pez, planters, price, pringle, rodeo, selma, skybar, slauson, snocap, starburst, sugardaddy, toffee, vanowen, whittier, wilshire

Servers will again be moved in batches, and no one should experience more than 20 minutes of downtime!


 
© 1996-2007, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
9:50 am

Twinkie web server being moved to new hardware

Posted (September 10th, 2007 at 9:50 am PST) by Justin@DH

The twinkie web server went unresponsive and isn’t coming up from a reboot. We are moving it to new hardware now, which is the fastest way to restore service in this case. Services are already starting to come back up on the new hardware. An upgrade will be posted when all configurations and services are restored. Our apologies for the inconvenience.

— UPDATE Mon Sep 10 11:46:37 PDT 2007 —

The move has been finished for a little while and services are running fine. If you have further problems with this server please contact support.


12:30 pm

One-Click Installs: WordPress updated to 2.2.3

Posted (September 9th, 2007 at 12:30 pm PST) by DreamHost

This new version is a pretty tame security/bugfix release, so it’s recommended for all.

WordPress 2.2.3 release info

You may upgrade within the One-Click Installs page under the Goodies tab of panel.dreamhost.com.


9:08 am

resolved – babyruth down

Posted (September 8th, 2007 at 9:08 am PST) by Steven

Earlier I found the babyruth server was unresponsive. Subsequent attempts to restore it to service remotely have failed. A technician has been called in on the case and is heading to the data center. I do not have an ETA for when service can be restored yet but will update this post with further information when available.

Update: The babyruth server was able to fully reboot but took longer than usual.


6:59 am

Fiero server hardware change.

Posted (September 7th, 2007 at 6:59 am PST) by Javier

We sorry to report that the fiero hosting server required a hardware change due to a hardware failure, we are currently failing over the server onto new hardware which should only take a few minutes, if you are still having issues by 8am pacific time please contact support


10:50 pm

More Server Moves!

Posted (September 6th, 2007 at 10:50 pm PST) by Patrick

We’re continuing with our spring cleaning (in September) with a few more server moves. The following servers will be moved from our Switch and Data datacenter to our Alchemy datacenter around 12:00 midnight PDT:

barranca, dumdum, gummi, nilla, babyruth, sake, smores, bounty, nougat, fc23, fxdollar

Estimated downtime for each server is right at 20 minutes.


1:26 pm

Discussion List Upgrade – 09.10.2007 @ 2PM PDT

Posted (September 6th, 2007 at 1:26 pm PST) by beth

On September 10th, 2007 at 2pm, we will be performing an upgrade on our primary discussion list server. During this time, the server will be off-line, and all discussion lists will be inaccessible. This process will take approximately two hours.

E-mail and files related to discussion lists (including configurations and archives) will not be lost. All discussion list e-mail will be queued until the upgrade is complete, and delivered at that time.


9:43 pm

MySQL Server Moves

Posted (September 5th, 2007 at 9:43 pm PST) by Patrick

Three MySQL servers need to be ‘re-organized’ in our Alchemy datacenter. This requires shutting them down for a brief moment. This will occur tonight around 22:30 PDT. The three MySQL servers affected are:

tweety, woodstock, and topogigo

This will affect a small number of users in the randy, spunky, and janky clusters. Check this space for updates.


11:21 am

WebMail login problems

Posted (September 4th, 2007 at 11:21 am PST) by brians

We are fully aware of the problems that you’re experiencing accessing WebMail. Our Admins are working towards resolving this issue. We sincerely apologize for any inconvenience this may have caused. Please check back for future updates.

Update: 11:53am: The time was incorrect on one of our webmail machines, causing it to expire sessions almost immediately. We’ve corrected the time and everything appears to be back to normal.

Thank you for your patience!


10:41 am

Delorean web server being moved to new hardware

Posted (September 3rd, 2007 at 10:41 am PST) by Justin@DH

The delorean web server is having issues booting, it is being moved to new hardware now. Sorry for the downtime, we are working to get the new server up as soon as possible. Check back for updates.

— UPDATE: Mon Sep 3 11:07:21 PDT 2007 —

The server has been replaced and is back up. If you are experiencing problems with this server please contact support.


10:27 am

RESOLVED! DNS caching server is offline…

Posted (September 1st, 2007 at 10:27 am PST) by Steven

One of our DNS caching servers has gone offline and this may cause problems on our network. We’re on the case! Please check back for further updates.

Update: A backup caching server is letting the one that took a dive have a nap. The network issues should be resolving themselves now. If you are still having issues seeing your sites, please file a support request.

Update:There were dns issues this morning causing problems with sending email. The original caching dns server that had issues before has been replaced and all is functioning normally again. Our apologies for the downtime.


 
© 1996-2007, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
3:35 pm

One-Click Installs: Gallery updated to 2.2.3

Posted (August 30th, 2007 at 3:35 pm PST) by DreamHost

This update adds no features but does fix up critical security issues with the WebDAV and Reupload modules within Gallery. It’s highly recommended that you either upgrade or disable those modules as soon as possible.

Gallery 2.2.3 Release Info

You may upgrade by logging in to panel.dreamhost.com and going to the One-Click Installs section under the Goodies tab.


4:06 pm

RESOLVED Major Network Outage

Posted (August 29th, 2007 at 4:06 pm PST) by DreamHost

It seems our core router is having some major issues right now. We’ve got admins working on this and will post more info when we have it.

Update 4:12pm by Jamesh: The networking issue seems to be resolved, the outage should have only been for fifteen minutes or less. There are still some servers that may need attention, so please continue to wait to contact support if you’re having trouble.

Update 4:26pm by Jamesh: All webservers appear to have recovered gracefully, we all apologize for the outage.


4:02 pm

RESOLVED Emergency webmail maintenance.

Posted (August 28th, 2007 at 4:02 pm PST) by jordan

We are currently experiencing some strange behavior with our webmail service. At this time we will be performing emergency maintenance on a few of our webmail servers. No downtime is expected , but there is a chance that there will be a few short outages. We apologize for the inconvience and this post will be updated as soon as the webmail changes are complete.

Update 7:05pm by Jamesh: We finished fixing this around 6:35pm this evening, since then everything looks ok. If you’re still having any trouble with email or webmail please contact support with details.


12:14 pm

Network Problems – router issues persist

Posted (August 27th, 2007 at 12:14 pm PST) by Ralph

Sorry about the network problems. As has been documented on DreamHost status we have been dealing with network issues lately. We are on top of this problem right now and we’re working on getting things back up for you. We are in continued contact with Cisco support and we have outstanding tickets with them regarding these issues. Please check here for updates but we’ll do our best to get things working as soon as possible.

Update 8/29 4:50pm: We are still working with Cisco on this issue but it is not actively causing any problems. When the router does act up we will post new announcements for those cases. So even though this is an ongoing issue, the problems themselves are not ongoing unless you see a post from us regarding a problem. Just wanted to clarify.


2:33 am

RESOLVED ouzo down

Posted (August 26th, 2007 at 2:33 am PST) by DreamHost

We were forced to powercycle the server a few times and finally got everything backup. We’ll keep a close eye for related issues. If you’re still having problems please contact support.


9:19 am

RESOLVED Topanga being moved to new hardware

Posted (August 25th, 2007 at 9:19 am PST) by Steven

Just minutes ago a drive on topanga went south. The server is being moved to new hardware at this time. Please check back for updates!

Update: The move to new hardware has been completed. Most sites are up and running but there were some errors that are still being worked on to restore service for other sites. I’ll post another update once everything is 100% with topanga again.

Final Update! All configuration updates have finished running and all sites should be good to go now. If you’re hosted on topanga and experience any trouble with your site please contact support for further assistance.

Thank you for your patience.


1:36 am

Networking Problems

Posted (August 22nd, 2007 at 1:36 am PST) by jordan

We are currently experiencing networking problems similar to the one on August 16th, 2007. It is being worked on at this time and should be fixed asap. We apologize for the inconvenience

UPDATE – The networking issues have been solved for the time being but this post will be left open pending a more permanent fix.


1:04 pm

Webmail Changes

Posted (August 21st, 2007 at 1:04 pm PST) by micah

We will be migrating webmail to a bigger and better load balancer this evening. Webmail will be up on both its old (current) IPs and the new IPs at the same time so there shouldn’t be any downtime related to this change.

The move is scheduled to begin tonight at midnight, Wed Aug 22 00:00:00 PDT 2007 and should take approximately 1 hour for the actual migration. The DNS updates (updating ~1 million A webmail.domain.com domain records) might take up to 48 hours after that.

If you have any questions or are experiencing webmail issues after midnight tonight, please contact our support staff.

** UPDATE
The webmail change is being moved to Thursday Aug 23 @ 00:00:00 PDT 2007 . Please check back for more information at that time.

** UPDATE Aug 23 @ 00:12 PDT
Webmail is being moved at this time, there should not be any downtime during this move tonight and it make take time for you to be on the new load balancer due to dns propagation. If you are having any problems using webmail please contact support.

** UPDATE AUG 24 02:56 PDT
Webmail has been successfully moved over to the new load balancers. Dns may still be propagating for some domains, this will not cause any outage as it is still working on both new and old load balancers.

*** IMPORTANT INFORMATION FOR CUSTOMERS USING NON-DREAMHOST NAMESERVERS **
If you are doing the dns for your domain elsewhere , please update you webmail record to point to 208.97.187.139 . The old ip address will stop working in 2 weeks so please update your records asap.


6:26 am

RESOLVED! Tib hardware failure

Posted (August 18th, 2007 at 6:26 am PST) by DreamHost

The mysql server “tib” suffered a hardware failure around 1 hour 30 minutes ago, at 5am Pacific time. It rebooted itself, and spent the next hour checking its backup drive, only to hang. It will not remotely reboot so I am heading in to diagnose the problem! Estimated further downtime is one hour.

Update 7:33AM Pacific: Unfortunately, it turned out to be checking the raid volume, and fsck was eating the files. I am moving the backup drive to another server right now and will begin the restore process.

Update 945AM Pacific: The backups took a little coaxing, but they are now restoring as fast as the drive will read! I’m not certain exactly how long it will take, but your databases will come online as they are restored. Ballpark time to completion is 2 hours.

Update 1251PM Pacific: Looks like the backups finished restoring at 12:17pm. Please write technical support if you are having any other problems with your “tib” databases!


7:39 pm

DONE! Emergency router upgrade

Posted (August 17th, 2007 at 7:39 pm PST) by DreamHost

Tonight, hopefully starting at 10:30pm, we will be performing an emergency IOS upgrade to one of our core routers. This is to hopefully quench the problems we have been experiencing once and for all. The downtime is estimated to be no longer than 20 minutes, however if anything goes wrong, it could take much longer.

The impact of this upgrade is that it will bring many of our internet uplinks offline, and make a good portion of our network completely unreachable until the router comes back online.

Update 23:27 PM: We reloaded the router with the latest revision of firmware per Cisco’s (And our CCIE consultant’s) suggestion. All seems to be well, the router is passing traffic and the servers appear to have recovered gracefully.


 
© 1996-2007, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
12:52 pm

RESOLVED DNS Outage Identified

Posted (August 17th, 2007 at 12:52 pm PST) by Dallas

We have identified a runaway process that has been erroneously eating domain DNS records and that has been killed. The missing DNS data is being restored as quickly as possible now so any domains still down should be restored in the next couple of hours. We have already pushed out new software with safeguards against this situation happening again. The network intermittence yesterday caused database connection problems which triggered a race condition in our code.

We will be checking all domains for missing DNS records and restoring all of them so there is no need to contact us to let us know your domain is down. We will update this site when we believe all domains should be working again.

UPDATE @ 2:39PM PDT

Okay, we believe all domains DNS should be regenerated and working again now. Please contact support if you have any problems still!


12:38 pm

DreamHostStatus.com Back up!

Posted (August 17th, 2007 at 12:38 pm PST) by Dallas

The secondary network DreamHostStatus.com (this site!) is hosted on was saturated due to a large increase in DNS traffic. After working with our provider for a few hours we’ve got it back up and we’re working to make sure the network saturation does not return.

Now to explain the DNS Outage itself.


2:36 pm

RESOLVED Network Problems

Posted (August 16th, 2007 at 2:36 pm PST) by Ralph

Unfortunately we are experiencing a return of similar network problems that we experienced yesterday. Please keep your eye on this post as we will update it as soon as we have more information. Sorry for the inconvenience.

– UPDATE –

The problem has been tracked down to a naughty switch and has been resolved. Due to the amount of traffic that dreamhoststatus.com is getting, we are leaving it static for the time being. If you are still experiencing problems, please contact support.

UPDATE TO THE UPDATE -Thu Aug 16 16:38:09 PDT 2007-
Shortly after the last update, things went down hill again. It turns out the switch wasn’t the problem. For the time being, everything is back up and running and the problem appears to be fixed. Continue to check this space for updates and as always.

Thu Aug 16 16:51:46 PDT 2007
While the networking issues appear to be gone, we are still dealing with fallout from them. This includes a few non-responsive caching DNS servers causing more slow down. We are bringing these back up now and everything should be smooth again soon

Thu Aug 16 20:28:38 PDT 2007
DNS is back up and running. Everything appears to be working fine. We will keep monitoring this issue throughout the night to catch any problems that may arise. If you have any problems whatsoever, please contact support.

Fri Aug 16 13:12:38 PDT 2007
After a core router config change we believe the network outages of Wednesday and Thursday of this week will not return. We will be digging deeper into the situation to determine why the issue appeared suddenly with no change on our part. This problem report will be left unresolved for awhile longer until we’re quite sure it is not resurfacing.


10:08 am

RESOLVED Webmail downtime

Posted (August 16th, 2007 at 10:08 am PST) by Justin@DH

The admins are investigating a mysql issue with webmail which is causing downtime. We are already on the case and it will be fixed up shortly. If any further information becomes available we will update this post, so check back for updates.

Update: The issues have been resolved and webmail is working normally again.

Addendum: We apologize for the recurring issues with webmail this week – it was a case of the database machine used for webmail running out of memory however it is not underpowered in that respect so unfortunately simply throwing more hardware at the problem will not make it go away. Our database expert has installed some additional logging tools and is keeping a close eye on that machine so that he can catch and correct the source of the issue.


11:50 am

RESOLVED Network Slowness

Posted (August 15th, 2007 at 11:50 am PST) by Patrick

We are currently experiencing some networking difficulties which is causing parts of our network to be inaccessible. This is affecting everyone in a random manner. We are working hard to fix the problem right now and will have an update for you soon.

–UPDATE– Wed Aug 15 14:33:01 PDT 2007

As of now the network is running at about 95%, we’re resolving the post for now while we continue investigating.

Update: Thurs Aug 16 11:45 PDT 2007

We forgot to update this with what happened. It seems we ran in to an issue with arp table corruption between two of our main routers. The tables had arp entries, but no associated mac addresses. This was causing around 50,000 packets per second of traffic, all destined for our LAX datacenter, to broadcast in our primary datacenter. The initial location of the problem threw us off track at first, however after figuring out that the traffic as all supposed to be going down the road, we were able to fix it quickly.


11:45 am

RESOLVED Webmail outage

Posted (August 14th, 2007 at 11:45 am PST) by kitchen

We are currently experiencing some issues with our webmail system and are looking into the issue right now.

We will post an update here when things are working again!

Sorry for the inconvenience.

Update: 12:04pm: It appears the database server that handles the session management for webmail decided to freak out. Our database engineers are working on the issue and we should have it resolved shortly!

Update: 12:36pm: The database server issue has now been fixed, and webmail should be functioning normally again! If you have any issues, please contact support!


5:35 pm

RESOLVED Webpanel Maintenance, August 15, 10PM PDT

Posted (August 13th, 2007 at 5:35 pm PST) by DreamHost

We will be performing some maintenance on our central databases beginning Wednesday, August 15 at 10PM PDT and ending 3 hours later at Thursday, August 16 at 1AM PDT. The expected downtime within this window is 2 hours. This will bring the DreamHost Webpanel, as well DreamHost-provided Squirrelmail offline.

UPDATE 12:59 AM PDT:

It looks like everything finished up without a hitch about an hour ago so all should be well again!


11:43 am

RESOLVED LAX connectivity issues

Posted (August 13th, 2007 at 11:43 am PST) by kitchen

We are currently experiencing some difficulty with our network links to our LAX datacenter. This is causing issues with the swarthy, looney, and frisky clusters, as well as dedicated servers.

Our network engineers are looking into this and we will have a resolution as soon as possible.

Sorry for any inconvenience.

Update: 11:58am PST: Our dns caching server for our LAX facility was down, which was causing hostname resolution issues, which was causing dns lookups to fail, and making lots of timeouts appear for smtp services. It’s back up now, and everything should be fine!


10:41 am

MySQL server nobnin

Posted (August 11th, 2007 at 10:41 am PST) by Steven

This server is currently experiencing an extended period of downtime until it can be manually rebooted from the data center. Unfortunately, I do not have an ETA for when service will be restored but please check back here for further updates.

Update: One of our admins is now at the data center and will have the server back in service shortly.

Final Update! All is well again with the nobnin MySQL server. If you are, however, experiencing trouble with your service do submit a support ticket.

Thank you for your patience.


8:18 am

filer trouble

Posted (August 11th, 2007 at 8:18 am PST) by Steven

One of our filers took the morning off and has just recently been kicked back into service (via reboot). This would have caused downtime for those hosted on tron, thorin, frigga, genki, wiggle, slaw, andor, lilac and perhaps a couple more. This also caused a little email back-up that is now clearing up.

All is well again but if you are experiencing any lingering issues with your service do not hesitate to contact us:
support@dreamhost.com

Thanks!


 
© 1996-2007, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
8:58 pm

Database server madcat down

Posted (August 10th, 2007 at 8:58 pm PST) by DreamHost

The database server madcat is currently down due to hardware problems. Our admins are working bring it back up right now, and we’ll post here when we have more information.

Update 3:27am: we’ve just brought madcat up, I apologize for prolonged outage.


7:24 pm

Moving webserver blowpop to new hardware

Posted (August 10th, 2007 at 7:24 pm PST) by DreamHost

The webserver blowpop has been having some networking issues and kernel panics, so I’m going to be putting it on new hardware. This should be done in less than an hour, I’ll post an update when I’ve finished.

Update (8:45PM PST): The failover is complete, and blowpop is now running on new hardware. If you run into any issues with your accounts or sites on this specific server, please contact support. -Christian


6:00 pm

Announcement List Configurations

Posted (August 9th, 2007 at 6:00 pm PST) by MikeP

The panel’s announcement list area (Mail->Announcement List) has some new changes where you can specify the redirect URL for where you want visitors to go after filling out the form. This will make it a lot nicer for configuring redirects rather than doing it within the HTML code of your website.

Don’t worry if you’re still using the old method from the actual form code, that will still work, but we highly recommend you use the panel for configuring redirects now.

If you see any issues with your announcement list related to the redirect changes, please contact support!


10:17 pm

Problems with webserver ikari

Posted (August 8th, 2007 at 10:17 pm PST) by DreamHost

Currently we are having some trouble with the webserver ikari and one of the file servers it mounts. Our admins are working to fix the issue right now, and will post updates here when they have more information.

Update: 12am PST: We’ve completed the repairs on the file server and are going to perform a few more tests to make sure things are ok. Expect an update shortly!

Update: 12:34am PST: Things should be up and running normally again on ikari. If you are still having problems, please contact support.


3:17 pm

Mysql on dedicated servers

Posted (August 7th, 2007 at 3:17 pm PST) by kitchen

While upgrading a package, it seems we broke mysql on dedicated servers. We are working dilligently to fix this issue right now and will update this post when we have fixed the problem.

We sincerely apologize for the inconvenience.

Update: 6:10pm PST: ok, looks like we got all of the mysqls taken care of. If you are still having issues, please write in to support and we will take care of them immediately!


4:07 pm

One-Click Installs: WordPress updated to 2.2.2

Posted (August 6th, 2007 at 4:07 pm PST) by DreamHost

WordPress has a new minor security/bugfix release out that shouldn’t affect themes or plugins.

WordPress 2.2.2 announcement

Log in here to upgrade WordPress or any of your other One-Click Installs:

Dreamhost Panel: One-Click Installs Page


9:39 am

Whopper downtime

Posted (August 4th, 2007 at 9:39 am PST) by Steven

An admin is currently enroute to the data center to manually restart the server. Unfortunately, I cannot provide a precise ETA at this time but will post an update once more information is available.

Update: Unfortunately, it looks like there is a hardware issue. We are currently moving it over to new hardware. Another update once this is resolved!

Final Update: The move to new hardware went well. All that’s left are some configurations which are expected to take no longer than 30 minutes. If you’re experiencing any lingering issues with your service please contact support!

Thank you for your patience.


11:17 pm

Quick database modification at around 2am PST

Posted (August 2nd, 2007 at 11:17 pm PST) by kitchen

Around 2am PST we will be making a quick modification to one of the database tables the web panel relies on, which cause the web panel to be unavailable for a short period of time.

We expect the outage to take at most 30 minutes, but hopefully no more than 5.

We will update this post with more information when the upgrade occurs!

Update: 2:03am PST: and it begins… we’ll post again when it’s back up!
Update: 2:17am PST: and it’s finished! It took exactly 13 minutes 37.73 seconds to complete. Thank you all for your patience!

– The Happy DreamHost Database Optimization Team


4:27 pm

One-Click Installs: MediaWiki updated to 1.10.1

Posted (August 2nd, 2007 at 4:27 pm PST) by DreamHost

MediaWiki has released a minor bugfix and compatibility update. You can read about what’s new and decide whether or not to upgrade for yourself:

MediaWiki 1.10.1 Changelog

Log in here to upgrade MediaWiki or any of your other One-Click Installs:

Dreamhost Panel: One-Click Installs Page


3:32 pm

Filer Upgrade

Posted (August 2nd, 2007 at 3:32 pm PST) by micah

Tomorrow morning we will be upgrading the controller (head-unit) for one of our filers that has been causing problems lately. This will affect customers on the web servers monroe and sanborn.

This upgrade will take place at 7am PDT Friday, August 3, 2007. It should only take about 30 minutes to swap in the new head-unit.

We apologize for any inconvenience this causes but we’re confident this will greatly improve the stability for customers in the future.

Update: 8:20am PST: this is now finished. Things should be running normally again on monroe and sanborn. If you have any issues, please contact support.

Thanks!


 
© 1996-2007, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
3:20 am

randy filer on the fritz!

Posted (August 2nd, 2007 at 3:20 am PST) by kitchen

The filer ‘squeezer’ in the randy cluster has gone down and will be causing intermittent issues on machines in the randy cluster.

Our hardware team is looking into it right now and we will have an update for you shortly.

We apologize for any inconvenience!

Update: 3:27am – Squeezer is back online and kicking!


12:14 am

trans-am getting DDoSed

Posted (August 2nd, 2007 at 12:14 am PST) by kitchen

The server trans-am is under a heavy denial of service attack right now. Our network engineers are looking into it and we will have an update for you soon!

Thank you for your patience and we apologize for the inconvenience!

Update: 12:36pm PST: the ddos has been taken care of and trans-am is back in business! If you have any additional problems, please contact support.

Thanks!


8:39 pm

Moving milk onto new hardware

Posted (August 1st, 2007 at 8:39 pm PST) by DreamHost

Webserver milk has been kernel panicking lately, and so we’re going to move it onto new hardware. This process should take less than an hour and cause only brief downtime for those customers hosted on milk.

Update: 8:44pm due to unforseen difficulties this is being delayed until 10pm or so.

Update: 11:46pm – James apparently forgot to update this post. milk is all moved over to the new hardware and everything is running fine. Please contact support if you have any issues!


11:58 pm

stewarts getting new hardware

Posted (July 31st, 2007 at 11:58 pm PST) by kitchen

The machine ‘stewarts’ looks to have a faulty network card which is causing major instability in the machine. We are going to move stewarts over to new hardware.

This process will take about 30 minutes to an hour. We will post an update here when it has been completed.

Thanks!

The Happy DreamHost Night Owl Team.

Update 12:57am PST: the new hardware is online and services are up. If you have any problems, please contact support!


7:01 pm

Stimpy lost a drive

Posted (July 31st, 2007 at 7:01 pm PST) by DreamHost

The MySQL server “stimpy” has experienced a drive failure. We’re working on getting the issue resolved as soon as possible, and this entry will be updated when stimpy is back in action.

As always, if you have any questions about this, please contact support.

Update Tue Jul 31 20:59:11 PDT 2007: As things turn out, stimpy suffered from unrecoverable filesystem corruption on one of its data drives. The data is in the process of being restored from backup, and your databases will be back online as soon as we can. Further updates to follow.

Update Wed Aug 1 1:01am: The recovery process has been finished and stimpy is up and working again.


10:12 pm

Rack Consolidation!

Posted (July 30th, 2007 at 10:12 pm PST) by kitchen

In an effort to continue bringing you, our customers, our amazingly low prices, we’re consolidating some of our racks that are mostly empty, so we can reuse them without buying all new racks!

To do this, we need to move a few servers around. Tonight at around 2am we’ll be moving the following servers:
vito
crusty
destro
ouzo
baseline
zartan
snowdrop

The downtime per machine should be no more than 30 minutes, and we will update this post when the moves have been completed.

Update 2:03am PST: We are starting to move the servers. We will update this post when the moves have been completed!

Update: 3:27am PST: with the exception of snowdrop, everything is set. Snowdrop is in a different rack, so we’re doing that now.

Update: 3:41am PST: snowdrop is back up now.

All of the moves have been completed. Please contact support if you have additional problems! Thank you!


5:37 pm

Hummer getting new hardware

Posted (July 30th, 2007 at 5:37 pm PST) by Terri

We have not been able to get hummer to reply back to us, so we’re getting it on brand new shiny hardware. This should take less than an hour to complete. We’re very sorry for the downtime, but we’re going to get it back running very quickly.

Update (6:53PM): The failover process is complete, and the server is now operating on new hardware. Everything appears to be working normally now. If you encounter any issues with your sites or related services, please contact support. -Christian


3:20 pm

Sylvester getting new hardware

Posted (July 30th, 2007 at 3:20 pm PST) by Terri

The shared MySQL database server sylvester is breathing its last, and right now an admin is working very quickly to get another machine online to take its place. Expected downtime for sylvester database users is around two hours. We will update this post as sylvester’s replacement progresses and when it is back online. We apologize for the inconvenience.

Update Mon Jul 30 17:53:02 PDT 2007: The move is complete, all databases on sylvester are back online.


11:18 am

Mail Problems on Postal/Spunky/Randy/Spacey

Posted (July 27th, 2007 at 11:18 am PST) by Ralph

Unfortunately our primary mail load balancer crashed and rebooted causing mail problems to all the mail servers listed above. Our secondary load balancer took over but needed a manual update to get going. Everything is coming back now and we’ll keep an eye on the load balancer.

To see if you are on these mail servers log into the panel and click on the Account Status link in the top right. Under Email Server you will see what server handles your mail.


9:58 am

Skittle getting new hardware

Posted (July 26th, 2007 at 9:58 am PST) by Justin@DH

We are having some networking issues with the skittle web server this morning, it is being moved to new hardware so resolved the issue as fast as possible. Service will be restored shortly, check back for updates.

UPDATE

The move is finished, all services restored according to the monitoring system. Sorry for the downtime.


 
© 1996-2007, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
10:25 pm

Moving servers tonight

Posted (July 25th, 2007 at 10:25 pm PST) by Terri

In order to make more efficient use of power in our datacenter, I’ll be moving a few servers tonight between 2 and 3 a.m., PDT. The racks are very close to each other, so downtime per server should be no longer than five minutes each.

The servers being moved tonight:

paramount
holt
fairfax
whopper
joust

I will also be moving the MySQL machine snowdrop. Shared MySQL machines are a bit bulkier, so it may take a few minutes longer to move it, but I will be as quick as I possibly can!

Thank you for your understanding!

Update 3:19 a.m. PDT: Everything but snowdrop is all moved and up and running. Snowdrop’s move will be scheduled for a later time.


4:36 pm

Spacey mail problems

Posted (July 25th, 2007 at 4:36 pm PST) by Ralph

If you are on the spacey cluster you are possibly seeing mail send to you bounce back with a mail loops back error. We are working on this issue right now and the fix is currently running. You can tell if you are in the spacey cluster by logging into the DreamHost web panel and then click on Account Status on the top right. The area that says ‘Your Email Server:’ you will see what server you are on. If it’s spacey then you are affected by this. We are currently reconfiguring our backend database for mail service. This unfortunately takes a while to run but as it runs accounts are getting fixed. Sorry about the problems this has caused you we will have this fixed up for you soon.

The problem came up as a result of some security changes we were making on our mail servers.


3:54 pm

Postal file server moves

Posted (July 16th, 2007 at 3:54 pm PST) by DreamHost

On tuesday, July 17th, 2007, at 10pm, we will be moving the Postal file servers in to join their web server’s in our other datacenter. This will mean a total downtime of all web and email services for the postal cluster. We expect the downtime to take about 1 hour.

Edit: One day we will get the date correct.

All done! Everything should be back up and running now. Please contact technical support if you are having problems!


11:40 am

Replacing hardware on Thunderbird

Posted (July 16th, 2007 at 11:40 am PST) by brians

Our Admins are in the process of moving the machine over to new hardware. The machine should be back online within the next 30 minutes. We sincerely apologize for any inconvenience this may have caused. Please check back for future updates.


8:39 am

Dawber getting new hardware

Posted (July 16th, 2007 at 8:39 am PST) by Justin@DH

The dawber server is having issues booting, it is getting new hardware now. Service will be restored shortly, check back for updates.


8:25 am

Swarthy move issue

Posted (July 14th, 2007 at 8:25 am PST) by DreamHost

Seems we hit a slight bump with the move last night. Users on a particular file server currently can’t see their sites or their users due to this. We are in the process of fixing this right now. Updates soon.

Affected servers: andor atomic cyrax dali dimarco frigga fud genki gradius hug kenobi lilac limbo-swarthy narcissus noid postac rock samus sektor slash slaw soda squirm strange thorin tron van vidar vlad wiggle zebes

Update: 10:17am All the websites should be back up but still fixing email issues.

Update: 10:25am Email is now working.


9:43 pm

Swarthy Server Move Reminder!

Posted (July 13th, 2007 at 9:43 pm PST) by micah

This is a duplicate post to remind everyone that we’re moving a couple dozen servers to our LAX facility tonight:

Due to space and power constraints in our primary datacenter we are moving the entire shared cluster “swarthy” to our LAX facility at 11:00PM PDT, this Friday night, July 13th 2007. All webservers, mail servers, file servers, and MySQL servers in the swarthy cluster will be unreachable during the move, which we expect to take approximately 5 hours.

The swarthy cluster contains the following web servers: andor atomic cyrax dali dimarco frigga fud genki gradius hug kenobi lilac limbo-swarthy narcissus noid postac rock samus sektor slash slaw soda squirm strange thorin tron van vidar vlad wiggle zebes

If your site is on any of these servers this status post applies to you. Also, if you check the “Account Status” button on the upper right hand of the webpanel ( https://panel.dreamhost.com ) you should see your cluster under the section “Your Email Server:”.

We do apologize for this inconvenience. Rest assured we will do everything possible to keep downtime to a minimum.

If you have any questions regarding the server move, please contact our support team.

Update: 4:20AM: Most if not all services have been restored. Please contact support if your site is still down.


8:59 am

Century getting new hardware

Posted (July 12th, 2007 at 8:59 am PST) by Justin@DH

Century server is having some hard disk errors and isn’t rebooting cleanly, it is being swapped to new hardware now. Sorry for the downtime, we’ll have it back up shortly. Check back for updates.


4:25 am

More server moves!

Posted (July 12th, 2007 at 4:25 am PST) by DreamHost

With our continued concerns of space and power, we will be moving the following servers tomorrow morning, Friday July 13 beginning at 1AM PDT. The moves are expected to take no more than 30 minutes for any one server and should all be completed by 3AM.

bart bitters carl dawber gimli gir glass harpo homer larry nelson otto pogo raiden salt shemp smithers tak truffle yerba zim

If your sites are on any of these servers, they will be off-line while the move takes place.

Edit: All moves completed by 2:50AM PDT.


5:34 pm

RAM Upgrades!

Posted (July 11th, 2007 at 5:34 pm PST) by Patrick

The following three servers will have their RAM upgraded within the next 15 minutes:

zim, barranca, rosecrans

zim and barranca both need their BIOS flashed before they will accept the new RAM. This means about 10 minutes of downtime for each. rosecrans should only be down for about 5 minutes.

Check this space for updates.

– UPDATE Wed Jul 11 18:07:43 PDT 2007 –

All servers have been upgraded and are back up and running!


 
© 1996-2007, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
12:22 pm

Veyron getting new hardware

Posted (July 11th, 2007 at 12:22 pm PST) by Justin@DH

The veyron server has been having some hard disk issues and is being replaced now. Service will be restored shortly, check back for updates.

— UPDATE —

The server is back up and good as new!


10:50 pm

Emergency server moves

Posted (July 9th, 2007 at 10:50 pm PST) by DreamHost

The following shared hosting servers will be taken off line for a short period between 1AM and 2AM PDT, July 10:

fletcher firestone lifesaver tampa

The move should not take more than 15 minutes per server. If your site is hosted on one of those 4 servers, it will be off line during the move. We apologize for the short notice.

Edit: Finished as of 1:45AM


4:35 pm

Swarthy Server Move!

Posted (July 9th, 2007 at 4:35 pm PST) by micah

Due to space and power constraints in our primary datacenter we are moving the entire shared cluster “swarthy” to our LAX facility at 11:00PM PDT, this Friday night, July 13th 2007. All webservers, mail servers, file servers, and MySQL servers in the swarthy cluster will be unreachable during the move, which we expect to take approximately 5 hours.

The swarthy cluster contains the following web servers: andor atomic cyrax dali dimarco frigga fud genki gradius hug kenobi lilac limbo-swarthy narcissus noid postac rock samus sektor slash slaw soda squirm strange thorin tron van vidar vlad wiggle zebes

If your site is on any of these servers this status post applies to you. Also, if you check the “Account Status” button on the upper right hand of the webpanel ( https://panel.dreamhost.com ) you should see your cluster under the section “Your Email Server:”.

We do apologize for this inconvenience. Rest assured we will do everything possible to keep downtime to a minimum.

If you have any questions regarding the server move, please contact our support team.

Update: 4:20AM: Most if not all services have been restored. We are working on any straggling problems, please contact support if your website is still down.


8:12 am

Continental getting new hardware

Posted (July 9th, 2007 at 8:12 am PST) by Justin@DH

The server ‘continental’ appears to have some hard disk issues, and it isn’t coming up from a reboot. A replacement server is being configured right now and service will be restored shortly. Sorry for the downtime, check back for updates.

— UPDATE Mon Jul 9 08:31:40 PDT 2007 —

The move seems to have worked fine, all apaches are back up and the last of the configs are running now.


5:48 pm

Postal Filer Issues Causing Outages

Posted (July 6th, 2007 at 5:48 pm PST) by Jason C

One of our network fileservers on the postal cluster is currently having some problems. This is causing issues with a whole plethora of servers on the postal cluster. Members of our admin team are currently investigating the issue and we should hopefully have an update for you shortly.

We’re really sorry for any inconvenience this might cause you — but we swear that everything is being done to get things back up and running.

–Update Fri Jul 6 22:30:27 PDT 2007–

This was actually resolved about 4 hours ago, evidently we neglected to mark it so. Sorry for the oversight!


3:57 am

Some discussion lists down

Posted (July 4th, 2007 at 3:57 am PST) by Brian@dreamhost

One of our discussion list servers is currently having problems, and won’t come up via our normal, remote access channels. We’re currently working on getting the machine up ASAP and apologize for the trouble this may cause!

— UPDATE Wed Jul 4 08:52:05 PDT 2007 —

Our apologies for the downtime, the server is back up and working normally.


9:02 am

Webmail Down!

Posted (July 3rd, 2007 at 9:02 am PST) by micah

We had one of our central databases crash about half an hour ago. This database handles all of our webmail preferences and addressbooks which made webmail nonfunctional for everyone.

The server has been rebooted (it has lots of big drives so it takes a while to check the disks) and is all the way back up now.

We apologize for the inconvenience this may have caused. If you have any further questions, contact our support team.

Happy DreamHost Runnin-in-to-Downtime Team!


1:27 pm

Donate to Camp Laurel and Room to Read

Posted (July 1st, 2007 at 1:27 pm PST) by Josh

Tomorrow we’ll be changing the charities at our Charity Area, so it’s your last chance to donate to Camp Laurel and/or Room to Read and have your donation matched by us!

So far we’ve received 18 donations for $591.21 for Camp Laurel and 24 donations for $990.73 for Room to Read, not too shabby, but we can do better!


2:02 pm

galaxie getting new hardware

Posted (June 28th, 2007 at 2:02 pm PST) by kitchen

The server ‘galaxie’ appears to have some bad ram, and that’s causing it to crash frequently. Because of this, we’re moving it over to new hardware.

This process should take about 30 minutes to an hour, and we’ll post an update here once it’s done.

-The Happy DreamHost Kernel Panic Team

Update: 2:44PM PST: the switch has completed and services are up. If you have any problems, please contact support. Thanks!


11:28 am

Problems with email

Posted (June 28th, 2007 at 11:28 am PST) by danwy

We are currently experiencing some problems with our mail servers authenticating. Our admins are aware of the situation and are already hard at work to resolve it. We hope to have this situation remedied as soon as possible. Check back here for updates.

UPDATE 11:53am: Should be all back to normal now. One of our mail load balancing servers failed and was giving the errors while it was being switched over to a backup server. If you continue to have problems, please contact support.


 
© 1996-2007, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
10:17 am

Skylark getting new hardware

Posted (June 27th, 2007 at 10:17 am PST) by Justin@DH

The skylark server is having issues on startup, it is being moved to new hardware now. Sorry for the inconvenience, check back for updates.

— UPDATE Wed Jun 27 10:40:17 PDT 2007 —

The replacement went fine and services are restored. If you have further problems with this server please contact support.


12:15 pm

Cutlass being DDoS’d

Posted (June 26th, 2007 at 12:15 pm PST) by danwy

Some of you who are hosted on the ‘cutlass’ server may have noticed that your sites are unresponsive at the moment. This is because there is a large scale DDoS attack against multiple domains on that server. Our admins are working on it now and hope to have it back to normal shortly. We’ll update this post once we have more information.

–UPDATE– 12:35 pm: This should be back to normal now. We were able to divert the attack and the server load is well within an acceptable level now. If you have any further problems with your site(s) hosted on this server, please contact support.


5:10 pm

One-Click Installs: Gallery updated to 2.2.2

Posted (June 25th, 2007 at 5:10 pm PST) by DreamHost

Sorry about the delay on this one! This is a minor stability revision, and you can go here to determine whether or not you need to upgrade:

Gallery 2.2.2 Changelog

I’d post the upgrade instructions again, but you can just read them from the last post!


6:42 pm

One-Click Installs: WordPress updated to 2.2.1

Posted (June 22nd, 2007 at 6:42 pm PST) by DreamHost

The WordPress team released a new version with a couple bug and security fixes, which you can read more about here:

http://wordpress.org/development/2007/06/wordpress-221/

If you installed WordPress via our one-click installer, you can update to this latest version by logging in to panel.dreamhost.com, clicking on the Goodies tab and then going to the One-Click Installs page. You should see an “Upgrade to 2.2.1″ link to the right of your existing WordPress install. Make sure you back up all of your existing customizations before upgrading, as there is a chance they might be overwritten by the upgrade.


2:44 am

DreamHost Webpanel, now with less lag!

Posted (June 22nd, 2007 at 2:44 am PST) by michael

The long processes of database upgrades for the Webpanel is now complete. With a full round of upgrades of the database hardware and software, as well as countless hours of panel code and query optimization, the panel is now running much faster than it ever has. This is good news, since the panel is being used quite a bit more than ever before! It received 140,000 page views in just the last 24 hours. We will not rest on out laurels though! We know that we can do even better!


7:29 am

Unable to create new e-mail addresses in control panel. (RESOLVED)

Posted (June 21st, 2007 at 7:29 am PST) by David

Unfortunately we’ve got a problem somewhere that is generating errors when creating new e-mail addresses in the control panel. We apologize for the inconvenience this is causing. Our admins are currently working on resolving this problem as quickly as possible.

Please to not submit any support requests regarding this issue as we are already aware of it.

As soon as we have his resolved an update will be posted.

Thanks for your patience while we resolve this issue.

The not-so-happy DreamHost E-mail Creating Robot function :-(

RESOLVED 9:30am PDT – It’s fixed! Please try again and let us know if you run into any problems. If you’ve submitted a support request on this issue and haven’t received a response yet you might want to retract it. You can do that in the control panel under (SUPPORT > CONTACT SUPPORT) by clicking on the “X” delete link next to the request (at the bottom of that page).

The Happy-Once-Again DreamHost E-mail Creating Robot function :-)


12:03 am

Fresca down

Posted (June 20th, 2007 at 12:03 am PST) by Terri

Fresca suffered a hard drive failure, and is right now being transferred to new hardware. This should take no longer than an hour. We apologize for the inconvenience.

edited 12:36 a.m. PDT: Fresca’s services are all coming back up and sites are running again. Please submit a support request if are still affected by the server crash. Thank you for your patience!


10:43 pm

Maintenance Window Reminder

Posted (June 19th, 2007 at 10:43 pm PST) by micah

The following maintenance window will begin as scheduled at 11PM PDT:

http://www.dreamhoststatus.com/2007/06/15/unavoidable-power-restructuring-for-4-half-cabinets/

Happy DreamHost Power-Swappin Team!


5:37 pm

Server ‘yen’ getting new hardware.

Posted (June 19th, 2007 at 5:37 pm PST) by kitchen

The public network interface on yen failed, so we’re moving it over to new hardware.

This should take about 30 minutes, and when it finishes we will post an update here!

Update: Yen is over on the new hardware now, services will be coming up shortly.

Update: Yen is now back up fully. If you have any problems, please contact support!


7:38 pm

PHP/CGI permissions problems!

Posted (June 18th, 2007 at 7:38 pm PST) by kitchen

Many of you are still reporting issues with your php and cgi applications, and we pinned this one down, too.

On some servers, apache’s suexec binary was installed without setuid permissions, causing it to not function. This was causing user’s scripts to not run as their user. If the script needed access to certain files with strict permissions, it would fail. We’ve remedied this situation and sites should be working as usual now.

If you are still having problems with your sites, please contact support and we will look into them as soon as possible.

Once again we sincerely apologize for any inconvenience this caused.

Thanks,
The Happy DreamHost Workin Late On A Monday team!


 
© 1996-2007, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
4:50 pm

500 Internal Server Errors

Posted (June 18th, 2007 at 4:50 pm PST) by kitchen

We’ve gotten a flood of reports of 500 Internal Server Errors and have tracked down the cause!

We are upgrading a couple of packages and it’s causing php to break momentarily. For some reason when we rolled out the package last week to our “frisky” cluster, none of this happened, so we went ahead and pushed it out to everything today. Unfortunately, that’s causing some breakage while the new packages are being installed, and we are taking steps to prevent it from happening while we upgrade the rest of the servers.

What we thought was going to be a seamless upgrade turns out to be not so seamless. Rest assured that if we had known it was going to cause any downtime, we would have scheduled this for late night (PST), and warned everyone well in advance.

We sincerely apologize for any inconvenience this has caused.

Sincerely,
The Happy DreamHost BreakStuff Robot.


7:13 pm

SSL certificate renewal for most customers

Posted (June 17th, 2007 at 7:13 pm PST) by DreamHost

Our current “mail.dreamhost.com” certificate used by most of you for SSL email is up for renewal on Friday! We will be replacing it with a newer, better certificate with the name:

*.mail.dreamhost.com

It is also signed by an internal certificate authority, “New Dream Network Certificate Authority”. This means two things! First, most of you will get a popup saying “This certificate is new and untrusted! The world is ending!”. If you press CANCEL the connection will not go through, and you will have a chance to install our NDN CA certificate into your email client. This will allow your computer to trust us!

http://wiki.dreamhost.com/NDN_Certificate <----<<< link to installation instructions in our Wiki
https://dreamhost.com/ca/ndn.ca.crt <----<<< link to certificate file data

If you have any problems, please do not hesitate to contact us! This should make your email client not pop up a warning about how the certificate is from an untrusted or unknown certificate authority. This is different from the next (usual) warning about the certificate name not matching the host you are connecting to!

To get rid of the host mis-match error, all you need to do is head on over to https://panel.dreamhost.com/ and log in. In the upper right, click the link that says “Account Status”, then make note of your “email server”. You will want to edit your IMAP, POP, and SMTP servers to be something along the lines of:

a1.balanced.email-server.mail.dreamhost.com
For Example: a1.balanced.spunky.mail.dreamhost.com

Thats it! No more annoying popup windows when using SSL! It is now more secure than previously.

FAQ:
1. Why don’t these instructions work for apple mail?
-It seems there is a bug in apple mail. It does not properly use wildcard certificates. (*.mail.dreamhost.com should match any “word.mail.dreamhost.com”.) We will be contacting them on Monday regarding this issue. Remember, it’s not worse than it was previously!

2. Why not get a REAL certificate signed by VeriSign?
-This is a REAL certificate, and the SSL works just the same.

3. I don’t trust you, I have too many computers to do this on, I can’t expect my clients to install that CA certificate, etc, do I have to install the NDN CA certificate?
-No! Just click to accept the *.mail.dreamhost.com certificate permanently and it shouldn’t bother you until we renew or change the certificate. Installing the CA certificate would allow us to renew the certificate transparently.

4. Why don’t you set the content-type properly on the above link?
-It’s easier to save the file to disk for importing into Thunderbird, etc, if we don’t. Otherwise IE and FireFox both try to process the certificate.

5. What are the vital stats on the new certificate?

Certificate:
Data:
Version: 1 (0x0)
Serial Number:
e8:c8:92:78:d0:05:ce:5f
Signature Algorithm: md5WithRSAEncryption
Issuer: C=US, ST=California, L=Los Angeles, O=New Dream Network, LLC, OU=Security, CN=New Dream Network Certificate Authority/emailAddress=support@dreamhost.com
Validity
Not Before: Apr 12 00:48:57 2007 GMT
Not After : Apr 9 00:48:57 2017 GMT
Subject: C=US, ST=California, L=Brea, O=Dreamhost.com, OU=Security, CN=*.mail.dreamhost.com/emailAddress=support@dreamhost.com
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public Key: (1024 bit)

Modulus=E84958AF3CBFB6AA1060288E83E1B97CF75312B9EEBC194C71EB1A3A477706746134DFC
AD8539ACAA161284CA27C04E70DE479DB825E0EC1D5E0F479C380315F42D46304BE8D064458073
9A33D853A1B70CEF73C6389B09E31AA286B9031EC9CE68BEFBB8A6846E1F40AA6F34A218B5A72F
62C0A52B7B276998B909E344162FB

Update:
The certificate is installed, now, however some astute readers have alerted me to the fact that this new certificate isn’t actually X.509 specification compliant. We’re going to stick with it, since it does help a subset of our users, and will consider some alternatives for the future!


7:08 pm

Unavoidable Power Restructuring for 4 half-cabinets

Posted (June 15th, 2007 at 7:08 pm PST) by micah

Unfortunately we were informed this week that one of our datacenters is requiring us to switch 4 of our circuits from one power feed to another. This will take place on Tuesday night, June 19, at 11pm PDT. The circuit switchover should take approximately 5-15 minutes from start to finish. This downtime window is for the following list of web servers:

admiralty
aero
alondra
altoid
arrow
aviation
broadway
bubbleyum
butler
cahuenga
cienega
contra
cornelius
gage
garvey
herod
lankershim
larchmont
limbo-randy3
malt
manchester
mel
moorpark
napoleon
nash
rhod
rossmore
rum
rygar
shekel
tetris
toffee
ultima
vanowen
wasabi

There are also six shared MySQL servers which house databases for customers not necessarily affected by the above list of web servers. These are:

bill
figaro
jarvis
madcat
mimsey
nala

We realize this will be inconveniencing quite a few people and you have our apologies, we simply have no choice in the matter.

– UPDATE Tue Jun 19 23:20:59 PDT 2007 –

The power switch is complete and everything is back up and running. If you are having any problems with your service, please contact support.


8:33 am

Postal file server issue

Posted (June 12th, 2007 at 8:33 am PST) by Justin@DH

One of the file servers on postal which serves files for a small number of customers is having issues this morning, causing some domain downtime for certain postal cluster servers. Admins are on the scene and are working on this issue now. Possible effected servers are:

annie apu bart bitters brandine burns carl chalmers cheeky cletus dawber dib flanders gaz gimli gir glass gloop gotcha harpo homer kearny knocker larry lenny lisa marge mcclure millhouse monkey nelson ninja otto pogo raiden ralphie salt seamus skinner smithers tak wiggum willie yerba zim

Our apologies for the downtime, we are working to have this fixed and back up as soon as possible. Check back for updates.

— UPDATE Tue Jun 12 08:48:21 PDT 2007 —

The file server is back up and we are fixing any remaining issues with file servers. Sorry again for the downtime.


5:12 pm

DreamHost Webpanel database upgrade June 20, 10PM PDT

Posted (June 11th, 2007 at 5:12 pm PST) by DreamHost

We will be performing some maintenance on our central databases beginning Wednesday, June 20 at 10PM PDT and ending 4 hours later at Thursday, June 21 at 2AM PDT. The expected downtime within this window is 2 hours. This will bring the DreamHost Webpanel, as well DreamHost-provided Squirrelmail offline.

Update Thu Jun 21 00:41:31 PDT 2007: Upgrades have been completed.


4:23 pm

File Server Maintenance

Posted (June 11th, 2007 at 4:23 pm PST) by micah

One of our file servers has been having increasing stability issues lately, due to the severity of the problems it causes when it goes down we are performing a live upgrade of it’s firmware and software today. The upgrade should only affect 233 users total, during this time we will do everything we can to prevent it from affecting anyone else. The upgrade should take approximately 45min and we’ll begin at 4:45pm PDT.

Unfortunately, due to security concerns, we cannot list the affected users here.

We realize this is short notice and regret the inconvenience this may cause some of you. This upgrade is expected to make things a lot more stable for everyone, and not just the affected users.

Happy DreamHost Filer-Fixin Team!

UPDATE: Mon Jun 11 17:34:08 PDT 2007

The upgrade is complete. All services should be back to normal.


3:17 pm

Web Hosting Break-Ins, Security Update

Posted (June 11th, 2007 at 3:17 pm PST) by Dallas

This is a follow-up to our previously announced security breach.

Once we identified the original problem, implemented some immediate security tightening measures, and assessed the scope of the problem, we immediately notified our affected users. The best way to improve security is to share information about flaws like this and we hope that being public about this will help. Though we are a bit ashamed that such a thing has happened to us, keeping it a secret would ultimately cause even more harm. We appreciate all of the information everyone has provided to us regarding this situation.

In the ensuing discussion since we announced this to our users we have seen and heard evidence that similar attacks have been successfully made on several other prominent web hosting companies in the past few weeks as well, some of them much more serious than this attack on us. That makes it look like this may not be an isolated incident. If anyone at other web hosting companies has any additional information, we’d love to talk about it!

We have already taken some steps to improve our security:

1. We have already removed the ability to view your passwords from our web control panel. That feature was in there as a convenience to users that misplace their passwords frequently but we feel it is no longer safe. This may have been used to grab our user passwords as part of this breach but there is some evidence suggesting that is not the case.

2. A couple of unexploited and minor security flaws in our software have been closed up as a precautionary measure. Thanks to those that reported these to us.

3. We have tightened down web control panel user logins to make it much harder for a would-be attacker to gain unauthorized access to our web control panel while still staying out of the way of legitimate users. This is not the sort of feature you should ever notice but it is there working behind the scenes.

In addition, we are working on some back-end changes to greatly improve our ability to identify potential security problems such as this one and to improve overall system security. A security breach is a very good wake up call and we intend to learn all we can from it. These are in progress and will be implemented in the coming days and weeks.

1. We are developing an early-warning system to notify us promptly when unusual activity on our web control panel is detected. The first phase of this has already gone live.

2. We are developing documentation to help our users take advantage of all of the security tools we have available already. We already support SSL for all email transmissions and we support SSH and SFTP for account logins. In the Internet of today, you can never have enough encryption and we want to make that easy to use.

3. We are also making a number of other behind the scenes improvements to tighten things down. We’re surveying our code and our system setups to identify areas where security can be improved and implementing changes as quickly as possible. Many of these changes are small but together they will create a much more secure system than before.

We have been actively monitoring activity on our network and investigating any reports of suspicious activity. If you believe someone is accessing your account without your permission, please notify our support team as soon as possible with as much information as you can provide.

If anyone has additional questions or concerns feel free to ask and I’ll check back through the comments and answer them.


8:36 am

Major Ddos

Posted (June 9th, 2007 at 8:36 am PST) by micah

Major ddos this morning has left many sites (web and mail), inaccessible. The attack is being addressed and seems to be dying down on its own. We apologize for this disruption of service and any updates will be forthcoming. While this issue is addressed and resolved, there might be a few straggling domains/sites that we’re working on.

In the meantime, if you have any questions, please contact our support team.

Happy DreamHost Up-All-Night yet again Team!


5:12 pm

New Feature: Disallow FTP on Shell users

Posted (June 8th, 2007 at 5:12 pm PST) by Dallas

If you have users that you only use with SSH and/or SFTP you can now disallow FTP logins altogether from our web control panel. To do this, simply go here:

https://panel.dreamhost.com/index.cgi?tree=users.users&

Then edit the user you want to change, and toggle the ‘Disallow FTP?’ checkbox. Save the user and the change will go live in a few minutes.


12:53 am

guinness backup drive replacement

Posted (June 8th, 2007 at 12:53 am PST) by DreamHost

It looks like the backup hard drive in guinness has called it quits. We have to take the server down to replace it, as it is not one of the hot-swap raided hard drives that your live MySQL data resides on. I’m sorry for the inconvenience, the downtime should only be about 10 minutes.

Update! The new backup drive is installed, and service is back online. Fresh backups of your data will start running shortly.

And, I lied. It crashed while formatting the drive. I’m heading back downstairs to see what the problem is now!

The server is teasing me. By the time I arrived in the datacenter and attached a crash cart, the machine was alive and well. It was just being unresponsive to even pings for a little bit. I will continue to monitor the situation and fix any problems that arise.


 
© 1996-2007, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
4:08 pm

Security Breach

Posted (June 6th, 2007 at 4:08 pm PST) by Dallas

UPDATE: 2007/06/07 6:49PM PDT – We are in the middle a more thorough investigation and some new information has turned up. While we did detect some unauthorized access to our user web control panel, in at least some cases it looks like that may not be to blame for the compromised ftp accounts. In some isolated cases it appears that there may be security problems on end-user computers as well. If you have been affected by this, please do whatever checks on your own computer you can as a precaution. Our investigation is covering all possible attack points and this is one of the possibilities.

Also note that we now have confirmed information that these ftp account hijackings are happening on other web hosts as well and it looks very likely like there’s more to this situation than just the security problem we detected within our own system.

We are now forcing all of the affected users who have not yet changed their passwords to do so before they will be able to upload anything again. This is necessary so we can continue to monitor the situation and see clearly what’s going on.

Original post:

A very small subset of our user accounts have been compromised due to a security flaw in our web control panel software. We have already notified those of you affected directly via email, aside from dedicated server customers who are being notified right now. If you are not on a dedicated server and you have not gotten an email from us your account has not been compromised and is likely safe. It’s still a good idea to change your ftp and web control panel password as a precautionary measure.

The security flaw allowed the attackers to log into our customer web control panel with the access privileges of another user. From our web panel they were able to access individual user password information. The attackers also attempted to gain access to our central database and billing information but were ultimately thwarted in that attempt. No credit card information or customer personal information was obtained.


11:21 am

WebFTP upgrade

Posted (June 6th, 2007 at 11:21 am PST) by jeremy

In just a few short minutes, we’ll be upgrading our WebFTP system. During the upgrade, we’ll have to take the system offline so we can change the database layout. It should only be a few minutes until WebFTP is up and running again.

OK, everything appears to be working again. Please let support know if you find any issues with the new version.


12:12 pm

File server issues

Posted (June 5th, 2007 at 12:12 pm PST) by andrea

If you’re on the spunky server cluster (check it in the control panel under Account Status in the upper right corner, your mail server would be listed as spunky), then you may be experiencing slow access to web and email, and may be having issues with your files (missing or wrong ownership) or users. We’re looking into this right now. We appreciate your understanding and patience, and apologize for all inconvenience this is causing you.

Update at 12:32pm Pacific: This is now fix, we’re cleaning up. If you’re still seeing problems, please contact us with as many details as possible (domain name, username, files missing, site slow, etc).


3:33 am

Fileserver firmware upgrades

Posted (June 5th, 2007 at 3:33 am PST) by DreamHost

Tomorrow night, Wednesday, June 6th, 2007, at 12:00AM (Midnight) and Thursday, June 7th, 2007, at 12:00AM we will be performing an upgrade on one of our file servers. This is required to fix a couple of bugs which have been fixed in the latest release of code. The total estimated downtime for people on affected servers is around 30 minutes.

This usually means that only people who actually have a user on these file servers will experience downtime. Most of these servers will be completely unaffected.

Wednesday’s maintenance may affect users on these machines: dawber mcclure sack homer burns gir lenny zim harpo kearny willie ralphie lisa tak seamus brandine flanders nelson smithers bitters marge wiggum cheeky gotcha salt chalmers annie glass otto millhouse dib bart gloop yerba cletus larry skinner carl pogo monkey gaz gimli

Thursday’s maintenance may affect users on these machines: york limbo-spunky4 tampa archibald

—-

Wednesday’s maintenance went off without a hitch. It took slightly longer than expected, but all of the machines are recovering gracefully. So far no reboots have been needed. It took about 45 minutes, instead of the estimated 30. I’m sorry about that.

—-

Thursday’s maintenance is complete, however the web servers seem to be having trouble mounting the upgraded file server. We are looking in to this now. Stay tuned.

Update: 1:30AM. Looks like everything is back to normal with this file server. Sorry about the extended downtime on this one! It was being stubborn.


5:38 am

Web Panel Access Intermittent

Posted (June 4th, 2007 at 5:38 am PST) by David

Some of you may be experiencing problems accessing our control panel. Our admins are investigating this issue and hopefully will have it resolved very soon. Please check back here for updates.

UPDATE 7:45am – The problem has been resolved. We apologize for the inconvenience.


9:09 am

Web Panel Down

Posted (June 3rd, 2007 at 9:09 am PST) by kitchen

As you may already know, the web panel is offline. We haven’t determined the exact cause yet, but believe it’s a switch failure. Our admins are working on it right now, and will have an update for you shortly.

Other functions may be affected, such as webmail, however all other services should be online.

We hope to get this fixed as soon as possible, and will have an update for you shortly :)

Sincerely,
The Happy DreamHost Panel Breaking Machines.

Update: 10:18am PST: It wasn’t a switch after all, the load balancer in front of the web panel / webmail machines kernel panic’d and had to be rebooted.


9:06 am

Spunky Mail Authentication Problems

Posted (June 1st, 2007 at 9:06 am PST) by brians

We are experiencing issues with the database machine that handles authentication. As a result you may encounter login errors when attempting to access your mailbox. Our admins are diligently working towards resolving this matter. We sincerely apologize for any inconvenience this has caused. Please check back here for future updates.

UPDATE: This issue was resolved literally seconds after my initial post. Again, we apologize for any inconvenience this may have caused.


8:54 pm

Database server outage

Posted (May 31st, 2007 at 8:54 pm PST) by DreamHost

We are currently experiencing a network outage on the following database servers:

precious slinky einstein dynomutt nipper otis rintintin grimm gromit bandit droopy pluto triumph zero checkers

There is an admin in the datacenter investigating right now, and we hope to have them working again shortly.

Updated 9:59pm the admin has rebooted the effected switch and all of these database servers should be up now.


8:36 pm

Power redundancy and brief panel downtime

Posted (May 30th, 2007 at 8:36 pm PST) by DreamHost

Our previously-mentioned power infrastruture changes will begin within the next 30 minutes, at it’s previously scheduled time.

In preparation for these changes, the DreamHost Webpanel will be taken offline briefly. Expected downtime is about 10 minutes and will begin at 9:45PM.

Update Wed May 30 22:41:52 PDT 2007: Panel is good to go. Power maintenance continues.


8:17 am

Replacing hardware for server chalmers

Posted (May 29th, 2007 at 8:17 am PST) by Justin@DH

Machine should be up within the next 30 minutes. Sorry for any inconvenience this may have caused. Please check back here for future updates.

— UPDATE —
The server is running on fresh new hardware, and all seems back up. If you notice further issues with this one server please contact support.


 
© 1996-2007, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
4:09 pm

Spacey filer issues

Posted (May 26th, 2007 at 4:09 pm PST) by kitchen

One of our network fileservers on the spacey cluster is having some problems and causing issues with many servers on the spacey cluster. Our admins are looking into the problem and will have an update shortly!

We apologize for the inconvenience!

Update 18:24PST: Looks like one of the disk shelves blew a fiber channel controller. We’ve replaced the controller and are making sure it’s happy before we put it back into production. Update soon.

Update 19:38PST: The file server is back online and we are working hard to get services smoothed back to normal. If you’re still having issues, please contact support. Thanks!


11:39 am

webmail attachments not working

Posted (May 26th, 2007 at 11:39 am PST) by kitchen

It appears that attachments in webmail are not working. We’re looking into this problem and will have a fix / update for you shortly!

We apologize for any inconvenience!

Update 14:35PST: this issue has been resolved and attachments look to be working fine! Please contact support if you have any further issues.


1:27 pm

Power Redundancy Implementation

Posted (May 25th, 2007 at 1:27 pm PST) by Patrick

On Wednesday, May 30th at 10 p.m. PST, we will be adding redundant power to several racks used for infrastructure. Each rack has two power strips, both supplied by the same source of power. We will be switching one of those strips in each rack to an alternate source. The end goal is to have vital central machines be supplied by two different sources of power.

We will be switching over a total of 12 racks, one rack at a time. These racks include machines that our web control panel connects to, as well as some of the file servers that customer data is stored on. Since each machine in these racks has redundant power supplies, service should not be interrupted during this switch. Please check this space for updates.

– UPDATE Thu May 31 01:37:21 PDT 2007 –

All planned circuit switches have been completed!


12:39 pm

Filer reboot.

Posted (May 25th, 2007 at 12:39 pm PST) by mir

One of our filers on the spacey cluster was having problems and eventually crashed as our admin team was trying to fix it. This was causing very high loads on several servers in the spacey cluster and errors for customers trying to access their sites and scripts. The filer is back up and running now and we are rebooting the servers it is mounted to. This should take no more than 5-10 minutes after which things will be working properly again. We do apologize for this and are trying to figure out a way to prevent this from continuing to be an issue.

Update: we have rebooted all of the effected servers (a few apaches may take a few minutes longer to start up but we have all of the machines working now).


11:21 pm

More DNS Issues

Posted (May 23rd, 2007 at 11:21 pm PST) by Terri

Our DNS issues from earlier are back. The solution the load balancer vendor gave us worked for only a short time. We are working right now to find a solution until we can contact support again and resolve this for good. Until then reaching our network will have slower response times than normal.

We apologize for the inconvenience.

UPDATE–

After working with our vendor we seem to have the problem under control. We are leaving this post in an “unresolved” status still until we are 100% sure the problem is not coming back as it did yesterday but so far everything is looking much better.

UPDATE– Thu May 24 10:42:09 PDT 2007

Things certainly seem better. We’re leaving this post as “unresolved” for now while we troubleshoot the outage but as of this moment, DNS resolution appears to be working fine.

RESOLVED —
The problems regarding dns and our load balancer have been resolved. It was a combination of problems that caused the outage. The main problem was that the default udp settings on the load balancer did not have any timeout values set (BAD!!!). This was causing any udp connections that didn’t complete to fill up connection tables on the load balancer causing its CPU to spike and stop serving new requests. We sincerely apologize for this outage and are working on new plans that would prevent a similar outage in the future.


3:30 pm

DNS reliability issues

Posted (May 23rd, 2007 at 3:30 pm PST) by jeremy

For a short while now, our 3 main DNS servers have been experiencing some more troubles. We’re currently in contact with the load balancer vendor trying to get to the bottom of it all. We don’t currently have an ETA on when everything will be fixed, but until then you may experience intermittent issues connecting to your websites.


10:30 am

Dreamhoststatus maintenance

Posted (May 23rd, 2007 at 10:30 am PST) by DreamHost

There will be a scheduled maintenance tomorrow (5/24) starting from 9am PDT to 10am PDT. Dreamhoststatus will be unreachable during the maintenance. Sorry for any inconvenience.


2:46 pm

Panel database maintenance, Wednesday night

Posted (May 22nd, 2007 at 2:46 pm PST) by DreamHost

At Wed May 23 22:00:00 PDT 2007 we will be taking the DreamHost webpanel offline for some maintenance. The downtime window extends for one hour, til 23:00:00 PDT. Expected downtime within this window is 30 minutes.


1:52 pm

Network Problems

Posted (May 21st, 2007 at 1:52 pm PST) by Ralph

We are currently dealing with network issues that are causing some downtime for some of our servers. The problem involves the private network between web/mail servers with their file servers. This appears to be affecting all clusters on some level. We are fixing everything now so some issues have cleared up and everything is returning back to normal. For those of you who had SMTP problems and receiving problems, you were affected by these issues. SMTP should be up, but we’re still cleaning up the network issues which should be resolved shortly.

Update: It appears that everything has been fixed. We’re still investigating the cause, but it appears that one of our Cisco devices decided out of the blue to start bridging some of our localized traffic. We’ll post another update once we manage to ascertain the root of the issue.


1:49 pm

SMTP Problems

Posted (May 21st, 2007 at 1:49 pm PST) by Ralph

We had some problems with a backend service that caused SMTP service to go down. Total downtime was around 30 minutes. SMTP service should be completely restored now. We are working on ways to balance the service that failed so that we can avoid downtimes like this in the future.

Note: if you are having sending and receiving problems you are likely still dealing with issues related to the newest post about network problems.


 
© 1996-2007, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
5:00 am

Network problems this morning

Posted (May 20th, 2007 at 5:00 am PST) by jordan

We are currently experiencing some networking issues that may affect access to Dreamhost servers. The problem is being looked into at this time and we will post with updated information as soon as it becomes available.

UPDATE —
The problems with network have been solved. They problem turned out to be a large amount of dns traffic that was hammering our internal dns servers and slowing them down to the point of connections timing out.


10:09 am

SMTP Problems

Posted (May 18th, 2007 at 10:09 am PST) by Ralph

We had some problems with SMTP a while ago and the issue is now resolved. Sorry about the service downtime. One of the drives on the database server used for SMTP authentication caused the problems noticed today. We have fixed the issue and are taking steps to make sure this does not occur again, namely a monitor service on the drives for the server. Sorry again about the service disruption, you should now be able to send messages without any problems though there may be a delay initially.


8:48 am

Spunky Downtime

Posted (May 18th, 2007 at 8:48 am PST) by brians

We are experiencing problems with one of our network filerservers. This is causing inflated loads on machines in the spunky cluster. As a result, you may experience connection issues when attempting to access your site, email, etc. Our Admins are diligently working towards resolving this matter. Below is a list of the affected machines. Please check back here for future updates.

Updated list of servers:
alvarado archibald arrow blowpop bottlecap burton butler catalina cerritos crunch doheny fletcher fudge garland gobstopper gumdrop haven hillhurst hyperion lankershim lemonhead lucas manchester mills nash nilla ord plenty price pringle saltine schnapps starburst sunset taffy tequila western whittier york

Update! The above list of servers are the ones which actually had mounts on the file server in question. It seems we experienced some file system corruption on one of the volumes this file server mounted. Our attempts to repair the damage were not fruitful, so we have restored your home directories from backup. This required many reboots of various web servers and the file server itself. We apologize for the downtime this caused.

The data restore affects exactly 45 users data, outside of any downtime, so this most likely does not apply to you.

We have upgraded the file server head unit to the latest version of its operating system, so hopefully this will not be an issue in the future.


11:51 am

Replacing hardware on Espresso

Posted (May 17th, 2007 at 11:51 am PST) by brians

Machine should be backup within a hour ( worse-case scenario ). Sorry the inconvenience this has caused. Please check back for future updates.


11:07 am

PHP Security Updates: 4.4.7 and 5.2.2

Posted (May 16th, 2007 at 11:07 am PST) by Dallas

This was erroneously not previously mentioned. A number of critical security holes have been closed for both php 4 and php 5 and we are in the process of upgrading them on all of our servers, starting May 7, 2007. We have heard reports of a few problems with specific applications such as Drupal and WordPress (XMLRPC only). We normally go slowly with PHP upgrades do to their propensity to cause problems but the nature of the security flaws plugged by these versions has mandated a quicker than normal upgrade.

Servers in the postal, swarthy and randy clusters have been updated already. You can figure out your cluster by looking at the ‘Account Status’ box in our web control panel. The cluster name is listed as ‘Your Email Server’.


9:56 am

One-Click Install: WordPress updated to version 2.2

Posted (May 16th, 2007 at 9:56 am PST) by DreamHost

This is a pretty hefty feature upgrade which you can read more about here:

WordPress 2.2 Changelog

As always, you can upgrade your current one-click install of WordPress by logging in here:

https://panel.dreamhost.com/?tree=goodies.installer&

And then clicking on the “Upgrade to 2.2″ link to the right of your existing WordPress install. If you have any problems with this upgrade breaking your themes, plugins, etc, feel free to write in to contact support via the Support Page and put “Attn: Charlie – WordPress 2.2 Upgrade Problem” in the subject, and make sure to describe the problem as fully as possible.


2:07 pm

Ruby on Rails 1.2.3

Posted (May 15th, 2007 at 2:07 pm PST) by Terri

It’s that time again! Right now I’m compiling a new version of Ruby on Rails for our servers. I will be testing and slowly rolling these out this week and next week. The upgrade should be completed by the end of next week at the latest. I will be carefully testing the upgrade to be sure it does not impact customers negatively, but I highly recommend that everyone freezes their rails – especially if you are running a Typo blog!

To freeze your rails applications, ssh into the directory that holds the application and run rake freeze_gems. If you ever need to unfreeze the application, run rake unfreeze_rails.


10:46 am

Replacing hardware on Ruble

Posted (May 15th, 2007 at 10:46 am PST) by brians

Machine should be up within the next 30 minutes. Sorry for any inconvenience this may have caused. Please check back here for future updates.

UPDATE: Machine was back up within 15 minutes from my initial post


10:37 am

Panel Slowness

Posted (May 14th, 2007 at 10:37 am PST) by micah

One of our central databases is having some serious issues at the moment. This is likely causing the web panel to be somewhat slow while we address the problem. This only affects https://panel.dreamhost.com, all of your sites, email, and individual MySQL databases are unaffected.

We apologize for any inconvenience this causes.

UPDATE — A couple of code changes later and the MySQL server seems to be keeping up much better now.


9:01 pm

Looney Server Moves! (reminder)

Posted (May 12th, 2007 at 9:01 pm PST) by Patrick

Due to space and power constraints in our primary datacenter we are moving the entire shared cluster “looney” to our LAX facility at 12:01AM PDT, Sunday morning, May 13th 2007. All webservers, mail servers, file servers, and MySQL servers in the looney cluster will be unreachable during the move, which we expect to take approximately 5 hours.

The looney cluster contains the following web servers: apok axl blanka brazil charm curly dhalsim egon guile limbo-looney moe slappy soy tank tofu vega winston
If your site is on any of these servers this status post applies to you. Also, if you check the “Account Status” button on the upper right hand of the webpanel ( http://panel.dreamhost.com ) you should see your cluster under the section “Your Email Server:”.

We do apologize for this inconvenience. Rest assured we will do everything possible to keep downtime to a minimum.

– UPDATE –

Everything is back up and running in our LAX datacenter! Most services were restored around 3 a.m., but there were a few stragglers that had to be whipped into shape. If you’re having any problems with your account, please contact support.


 
© 1996-2007, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
3:36 pm

Support Contact Form Error

Posted (May 12th, 2007 at 3:36 pm PST) by mir

We are currently experiencing an issue with one of our internal databases which is causing issues with the support contact page on our web panel. Due to the nature of the database service in question we can’t just kill it off and restart it so we have our database expert working on it and will have it back up as soon as possible. We apologize for the inconvenience in the meantime and will update this post as soon as it is back up or we have more detailed information.

Actually already had an update while writing this post – the service is being restored now and should be fully operational within the next 45 minutes at most (it varies greatly and may be back within a few minutes as well so we’re giving the most conservative estimate).


12:06 pm

Filer down – causing high machine loads on our spunky cluster.

Posted (May 12th, 2007 at 12:06 pm PST) by mir

We are currently working on our of our central filer machines that has gone down – unfortunately I do not have details currently on the exact cause but our admin team is trying to get it back up as quickly as possible as it causes rather severe issues on any machines that have that have data from that machine mounted. As soon as it is back up everything will return to normal so we’ll post again as soon as we have more detailed information (we are expecting no more than an hour and hopefully much less for this outage).

Update: the filer is back up and appears to be fine, we are making sure all machines mounting data from that filer are working properly again and everything should be back to normal within the next 5-10 minutes (only two servers appear to have been down completely from this but we’re rebooting other servers that had high loads as a result of the outage).

Update: It seems that one of the fibrechannel cards was misseated/crashed in the file server. After reseating everything and replacing out a couple of adapters the machine booted up. All in all none of the hardware wound up needing replacing, but the items showing the fault lights were not what was broken, as is sometimes the case! I am not certain where the “two” number came from, it was closer to 12-20 machines directly affected. We believe most of them to be back online now. Perhaps a typo along the chain of information!


11:10 am

hardware failure on van

Posted (May 8th, 2007 at 11:10 am PST) by brians

We are in the process of replacing the hardware on the machine. Total downtime will be ~1 hour . We will update as the move finishes.

UPDATE: Everything has been resolved. The machine is now fully functional. We apologize for any inconvenience the downtime may have caused.


9:59 am

Network Problems

Posted (May 8th, 2007 at 9:59 am PST) by brians

Network link between two of our facilities are having issues. This affects all dedicated customers and machines in the frisky cluster. Our Admins are diligently working towards resolving this issue. Please check back for future updates.

barracuda belair boxster bugatti cadillac camaro carrera cayman chevelle corvette cutlass dart edsel eldorado elise enzo esprit fairlane ferrari firebird gremlin hornet hudson hummer impala manmeat mclaren merkur miata model-t murcielago nova pacer pinto scion shelby skoda stingray stratus studebaker tesla thunderbird touareg veyron

UPDATE: Had to do a quick hardware replacement. The Link is back up. Again, we sincerely apologize for any inconvenience this may have caused.

UPDATE 2: The hardware replacement was not on our equipment. Our network provider/colocation provider had a hardware failure on their 10Gbit link between downtown LA and their LAX facility. We’re already working on a backup solution with them and hope to have some extra redundancy in place shortly.


2:23 pm

New Support Form

Posted (May 7th, 2007 at 2:23 pm PST) by pete

Some of you may have already noticed, we have a new support form.

https://panel.dreamhost.com/?tree=support.msg

It still has some bumps that need smoothing out, and it does require javascript! Some tests results are still being refined, and we plan to add more in the future. Suggestions and comments are welcome.


8:53 am

Spacey cluster issues

Posted (May 6th, 2007 at 8:53 am PST) by Justin@DH

We are currently having hardware issues with a file server in the spacey cluster (you can check your cluster in the control panel by clicking the “Account Status” link in the top right, then looking at the “Your email server” section). This is only effecting a small number of machines but is causing some sporadic downtime. The admins are working on resolving this issue. Our monitoring system and the oncall admins are fixing up the problems resulting from the file server issue as they happen and as soon as possible to avoid downtime. Please check back later for updates.

— UPDATE —

After some software upgrades on the file server Sunday and watching it carefully, it is much more stable now and hasn’t been causing further issues. We will continue monitoring for further issues, but at this point the issue is resolved.


9:22 am

Server vine moving to new hardware

Posted (May 4th, 2007 at 9:22 am PST) by Steven

The server vine has been having some hardware issues so we’re going to move it over to new hardware. The process should take about 30 minutes. We will post an update here when the process is complete.

We apologize for any inconvenience, and thank you for your patience!


1:55 pm

Support Response Times Decrease

Posted (May 3rd, 2007 at 1:55 pm PST) by michael

The average support message response time has improved significantly for 2007 at DreamHost. Today has been especially good with almost immediate responses to all incoming messages. By continuing to grow the team at a very fast pace, while providing better and better tools to the support technicians, we have been able to continue to improve our team. We’d like to thank everyone for all the supportive and thankful messages we have been getting via the support system.


3:37 pm

DreamHost Central Database Upgrade

Posted (May 2nd, 2007 at 3:37 pm PST) by DreamHost

We will be taking the DreamHost WebPanel and several other central DreamHost services offline for a Database Software Upgrade. This will begin next Wednesday, May 9 at 10PM PDT. The downtime window will continue for up to 6 hours, until Thursday, May 10 2AM PDT, however the expected downtime within this window is only 2 hours.

This will take the following offline:
DreamHost WebPanel
DreamHost-provided webmail
DreamHost Wiki

Customer sites, databases will remain online, and customer email will be still be reachable via POP3 or IMAP.


4:14 am

Spacey cluster issues

Posted (May 1st, 2007 at 4:14 am PST) by DreamHost

We are currently suffering from some issues with the Spacey cluster and are our admins are actively looking into these problems. As soon as we have more information we will update and provide more pertinent information on the status page.

—————————————————————————————————————————————-

The file server issue has been resolved and everything should be returning back to normal.

If you are suffering from domain downtime please send in a support request.


 
© 1996-2007, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
5:31 pm

Looney Server Move!

Posted (April 30th, 2007 at 5:31 pm PST) by micah

Due to space and power constraints in our primary datacenter we are moving the entire shared cluster “looney” to our LAX facility at 12:01AM PDT, Sunday morning, May 13th 2007. All webservers, mail servers, file servers, and MySQL servers in the looney cluster will be unreachable during the move, which we expect to take approximately 5 hours.

The looney cluster contains the following web servers: apok axl blanka brazil charm curly dhalsim egon guile limbo-looney moe slappy soy tank tofu vega winston
If your site is on any of these servers this status post applies to you. Also, if you check the “Account Status” button on the upper right hand of the webpanel ( http://panel.dreamhost.com ) you should see your cluster under the section “Your Email Server:”.

We do apologize for this inconvenience. Rest assured we will do everything possible to keep downtime to a minimum.

If you have any questions regarding the server move, please contact our support team.


9:35 pm

hornet getting DoS’d, LAX facility suffers

Posted (April 29th, 2007 at 9:35 pm PST) by kitchen

The machine hornet is currently under a very large denial of service attack and it’s saturating our network to our LAX facility. Our network engineers are currently looking at the situation and should have a solution soon. We will post an update as soon as we have new information.

We apologize for any inconvenience.

Sincerely,
The Happy DreamHost Weekend OnCall Robot

Update 10:47pm PST: It’s all taken care of and everything should be back to normal shortly. Thank you for your patience :)

Update: 10:48pm PST: I spoke too soon. We will continue to investigate this issue and provide status updates when we can.

Update: 11:06pm PST: We seem to have the situation under control. We are going to continue monitoring things in case it starts back up again.


4:08 am

Bam going down for a quick ram upgrade

Posted (April 27th, 2007 at 4:08 am PST) by DreamHost

Bam seems to be a little short on memory. I am going to attempt to upgrade it now! Total downtime should be about 5 minutes.

The server is coming back online now with an extra gig of memory!


12:21 am

Emergency file server upgrade

Posted (April 27th, 2007 at 12:21 am PST) by DreamHost

The file server which has been a little crashy lately for users in the “spacey” cluster is being swapped out right now with newer-better-faster hardware. We’re sorry for the short notice, the server was on its way down when I realized that this post never made it live upstairs! The new server is booting up right now, and we hope service to be restored in a few minutes.

Okey dokey, the new server is up and running!


1:54 am

Couple of quick battery swaps

Posted (April 25th, 2007 at 1:54 am PST) by DreamHost

We need to swap the backup batteries in 4 of our file servers. Downtime per server should not excede 5 minutes. This will affect Spunky and Spacey users.

One completed!

Wed Apr 25 09:58:01 GMT [rc:notice]: The system was down for 201 seconds

Two! Two Completed!

Wed Apr 25 10:02:12 GMT [rc:notice]: The system was down for 212 seconds

Three!

Wed Apr 25 10:16:34 GMT [rc:notice]: The system was down for 229 seconds

Fourth and final!

Wed Apr 25 10:25:17 GMT [kern.syslog.msg:notice]: The system was down for 342 seconds

That last one the silly little battery connector cable did -not- want to go back on. :(


1:35 pm

Brief network downtime around 3:30PM PDT

Posted (April 23rd, 2007 at 1:35 pm PST) by jeremy

Updated: There was a little delay getting started, but after everything was ready, the switchover went very well. With the exception of a completely separate (and thankfully smaller) outage caused by a loose network cable, the planned work was completed with just a few seconds of downtime.

At approximately 3:30PM PDT this afternoon, we’ll be taking a small portion of our network offline so we can fix some packet loss issues that our provider has been unable to rectify. Everything will be pre-configured, so downtime should be less than 30 seconds.

Happy DreamHost Network Team!


4:17 pm

Server bundy getting new hardware

Posted (April 22nd, 2007 at 4:17 pm PST) by kitchen

The server bundy has been having some hardware issues as of late and we’re going to move it over to new hardware. The process should take about 30 minutes. We will post an update here when the process is complete.

We apologize for any inconvenience, and thank you for your patience!

Update: the services on the new hardware are coming up. Everything should be operational within a few minutes.

Update: Everything is up and running over on the new hardware. Please contact support if you are having issues. Thanks!


1:48 pm

DreamHost panel database maintenance

Posted (April 19th, 2007 at 1:48 pm PST) by DreamHost

Next Wednesday night, April 25th, we will be performing more maintenance on some of our central databases. This will take the DreamHost Webpanel offline starting at Wed Apr 25 22:00:00 PDT 2007 and potentially extending until Wed Apr 25 23:59:59 PDT 2007. The expected downtime within this window is 30 minutes.

00:15 PDT: New estimate 00:30 PDT for final resolution of the maintenance.


12:58 pm

Downtime on spacey cluster

Posted (April 19th, 2007 at 12:58 pm PST) by andrea

One of our file servers is getting rebooted. This may be causing slowness or downtime if your domains/email is on the spacey servers. No data was affected. You can verify if this affected you by checking your Account Status (Your Email Server) on the upper right side of the control panel.

We apologize for the inconvenience. Please check back for updates.


5:57 am

Connections Problems to the LAX Datacenter

Posted (April 19th, 2007 at 5:57 am PST) by Patrick

We are currently experiencing severe packet loss between the Garland building and our LAX datacenter. This is affecting most dedicated server customers as well as all customers in the “frisky” cluster. This includes the following web servers:

belair, boxster, bugatti, cadillac, camaro, carrera, cayman, chevelle, corvette, edsel, ferrari, gremlin, hornet, hudson, impala, manmeat, pacer, pinto, skoda, stingray, studebaker, touareg

We are investigating the cause now and will update you with more information as soon as it’s available.

-UPDATE 7:28 AM-

It appears this was the result of a massive Denial of Service attack on a customer in the LAX datacenter. The attack has been thwarted and everything is running normally again.


 
© 1996-2007, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
3:59 pm

Homer’s Gone D’oh

Posted (April 18th, 2007 at 3:59 pm PST) by Jason C

Just a heads up for everyone that is currently on homer. A member of our admin team is in the process of failing your server over to new hardware due to some issues with the system. This shouldn’t take very long for the admin in charge to fail the system over — and they’re working as fast as they can — but please be aware that your email should not be affected during this transition. I’ll be sure to post an update here when things are back up and running.

I’m mighty sorry about this, everybody.

Update: The server should be back up and running now! If any of you happen to see any issues on homer, please file a ticket in the web control panel and let us know what you’re seeing.


10:15 am

Slowness and downtime on spacey

Posted (April 17th, 2007 at 10:15 am PST) by andrea

One of our file servers is getting rebooted due to IO and mount errors. Those errors may have caused (or are still causing) slowness or downtime if your domains/email is on the spacey servers. No data was affected. You can verify if this affected you by checking your Account Status (Your Email Server) on the upper right side of the control panel.

We apologize for the inconvenience. Please check back for updates.


10:06 am

Partial network slowness

Posted (April 17th, 2007 at 10:06 am PST) by jeremy

The main distribution switch for one of our facilities is experiencing high CPU utilization which is causing a small amount of packet loss. Only a small subset of our users are currently affected and we’re working on determining the exact cause of the problem. We’ll post an update as we get more information.


12:43 pm

Central DB Issues – Panel Down

Posted (April 14th, 2007 at 12:43 pm PST) by kitchen

One of our central database servers decided to take a nap, and it’s causing the panel to malfunction. Our admins are taking a look at this right now and we will have a status update for you shortly.

We apologize for the inconvenience.

Update: The server blew a drive and caused the machine to freak out. We have rebooted the server and it’s currently replaying InnoDB journals. Things should be back up and running smoothly shortly.

Also, it appears webmail was affected by this as well. We apologize for the inconvenience.

Sincerely,
The Happy DreamHost Weekend Warriors Team.

Update: The database server has completed its startup sequences and seems to be functioning normally. Please contact support if you are still having issues.


7:45 pm

Possible outage tomorrow morning for a few servers

Posted (April 13th, 2007 at 7:45 pm PST) by DreamHost

We just got word from one of our datacenter providers that our LAX datacenter will be going to generator power tomorrow morning (Saturday April 14th) around 7:30am for a few minutes. There is no planned outage, just a very, very small chance of one for most of our dedicated servers, and shared servers who’s mail server is ‘frisky’. Again, right now we are not expecting any downtime for any servers, we just wanted to pass on this warning of a very small chance of one.

Update: This issue seems to have passed without incident, though there was much rejoicing.


7:39 pm

File server troubles

Posted (April 13th, 2007 at 7:39 pm PST) by DreamHost

A few servers are having trouble mounting one of the file servers they use. A quick list is amp, orangebang, ovaltine, tea, coffee, monster. We’re working to fix this right now, so if you’re on a server who’s name is drink themed please hold on.


1:13 pm

File server hardware swap

Posted (April 12th, 2007 at 1:13 pm PST) by DreamHost

One of our file servers for the spunky and randy clusters is experiencing watchdog timeouts. This means every couple of hours it is simply freezing up and rebooting itself! As you can imagine, thats not good, so we are performing an emergency hardware swap before the problem gets any worse. Your data, if on this server, will be inaccessible for about 10 minutes starting at 2:15pm.

We’re back online! The file server was swapped out, and the new one was reporting correctable ECC errors on one of the memory modules. We already swapped that out and the file server is back online. All services and servers should recover shortly. If you are still having problems in about 15 minutes, please let our technical support staff know!


3:35 pm

Power strip maintenance – Date moved!

Posted (April 9th, 2007 at 3:35 pm PST) by DreamHost

On Saturday we experienced the failure of a power strip:

http://www.dreamhoststatus.com/2007/04/07/downtime-for-several-servers/

Or so we thought! Turns out, that specific circuit has an open neutral. On Friday, April 13th Thursday, April 12th, at 9pm, we are going to be performing quick maintenance to fix this problem. Luckly, we likely will not have to turn off any servers to do this. There is one circuit we will need to turn off to work on the broken one, but it is feeding a rack of servers with redundant power supplies! If all goes well, you, the Happy DreamHost Masses, will not notice any downtime.

If the wiring is not how we think it is, however, we will need to turn off two extra circuits feeding these servers:

avalon burbank catalina cognac deblume euclid forint grizabella heaven jake jalapa jawbreaker mufasa nwtekno ocean pippo sawdust sawtelle scar temple venice yoda

Some of those are dedicated servers, others are database servers. If we do need to cut power to these, the downtime will be approximately 1 hour.

We will be sure to have ominous sounding music queued up for doing this on Friday the 13th.

Due to a scheduling SNAFU on our part we need to move this maintenance window UP one day to Thursday, April 12th at 9pm. Again, we believe we will not need to actually turn off any servers in this window, it is a “just in case” thing.

Update 9:32pm 4-12 by JamesH: This is effecting a few more servers than we thought, but should be done with shortly. We’ll post as soon as we have more news.

Update 10:15pm by Kelly: This is obviously taking longer than anticipated. The box was in worse condition than originally thought, but could not be determined until we opened it up. We are pulling fresh wire right now to the boxes. ETA is now 11:30pm-midnight.

Update 10:42pm by Kelly: The juice is flowing again, we are now working on getting everyones service restored!

Update 12:05am 4-13 by JamesH: I just got word that all servers are up and working. I apologize for the outage.


11:00 am

Network Hiccup

Posted (April 9th, 2007 at 11:00 am PST) by micah

Our core router’s CPU spiked a few minutes ago causing systemwide network inaccessibility and slowness for approximately 10 minutes.

Everything seems to be back to normal and we’re investigating the cause of the spike.


1:29 am

Major DNS DOS attack

Posted (April 8th, 2007 at 1:29 am PST) by micah

We are currently experiencing intermittent DNS outages on all of our nameservers, this is causing problems with sites and email functioning properly.

We think it might be the DNS load balancers we’ve recently installed but haven’t determined that for sure yet. More information will be posted as we have it.

Our apologies for the inconvenience this causes.

UPDATE (2007-04-08 03:20 PDT): It appears we are the victims of an Easter DOS attack against our nameservers, we have an admin at the datacenter and our network expert onhand to deal with it. Hopefully things will be back to normal shortly.

UPDATE (2007-04-08 04:54 PDT): We seem to have isolated and resolved the DoS attack for the moment. All of our DNS servers have been resolving domains correctly and without pause for the past 20 minutes now so it appears that this nasty episode is behind us. We’ll be mopping up and fixing servers that might’ve crashed or otherwise gone wonky in the interim. Again, our sincerest apologies for all the downtime.

Happy DreamHost Does-Not-Sleep Team!


 
© 1996-2007, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
4:57 pm

Downtime for several servers

Posted (April 7th, 2007 at 4:57 pm PST) by DreamHost

A powerstrip in one of our racks has failed, leaving the following webservers without power:

firestone
fletcher
bob
lifesaver

In addition the following database servers are also down

scar
grizabella

We have an admin in the data center working to fix this right now. I will post updates with any information as soon as I have it.

Update: 8:15pm 4-7-7. All the effected servers are up and functioning now. If you’re still having any problems with your sites or databases hosted on the above servers please contact support.


11:59 am

Jezebel : up and down

Posted (April 6th, 2007 at 11:59 am PST) by Steven

Our apologies for the downtime experienced with this web server. We’re working as quickly as possible to resolve the matter and restore server performance. Please check back here for further updates.

UPDATE: We are in the process of replacing the hardware on the machine. Everything will be resolved shortly. We sincerely apologize for the inconvenience this has caused and apprecitate your continued patience. Please check back for furture updates.

UPDATE:

I set Jezebel up on new hardware yesterday but unfortunately it was having problems this morning (totally unrelated to the initial case but we couldn’t consider things resolved) so I activated a crash logging system and kept top (to watch the processlist) all day and of course it has behaved. We’ll be monitoring it but I feel it is safe to say the issues with this machine are fixed.


7:31 am

Web server “ugh” is not in a good way. (RESOLVED)

Posted (April 6th, 2007 at 7:31 am PST) by David

We apologize for the inconvenience, but we’ve found that the web server “ugh” is having hardware problems. We are in the process of replacing it now. We should have it back up on new hardware within an hour or so.

Please do not open any support requests for this machine until we have it back up again.

Updates will follow.


Update 9:11 AM

Ugh should be back online now and all services restored. Apache seems to be running OK and the server is overall showing normal load. Please contact us via the Contact Support form is anything seems amiss! It’s been up now for at least 30 minutes, sorry about the delay in updating.


9:22 am

One-Click Installs: WordPress updated to 2.1.3, Gallery updated to 2.2.1

Posted (April 4th, 2007 at 9:22 am PST) by DreamHost

Here’s the update info:

Amazing WordPress 2.1.3 Update Info!
Amazing Gallery 2.2.1 Update Info!

For WordPress, it’s basically more exploit fixes and proofing, while Gallery 2.2.1 is a relatively minor bugfix release. Log in at the link below and click the upgrade links if you wish to perform these upgrades:

Dreamhost Panel One-Click Install Page

Be ready to restore your backup in case anything goes wrong, as sometimes happens with any software upgrade!

**UPDATE**

Ack, I messed up copying the new WordPress and Gallery binaries to some of our machines, so if you ran a One-Click upgrade between 10am and 2pm PST on Wednesday April 4th, and your WordPress or Gallery didn’t upgrade properly, write in to support@dreamhost.com with a subject of “Charlie messed up my One-Click Install!” with a list of one-clicks’ URLs. Sorry about that!


1:32 pm

Last chance to donate to MIT and Save Darfur!

Posted (March 29th, 2007 at 1:32 pm PST) by Josh

Just reminding everybody.. in the next couple of days we’ll be changing the charities we have up at our panel that you can donate to directly via your hosting bill.. currently MIT OpenCourseWare and SaveDarfur.org!

So far we’ve had 27 donations totalling $689.19 to MIT and 41 donations totalling $1694.45 to Save Darfur … we can do better!

And of course, all donations will be matched by us!


11:57 am

Backend Panel Maintenance Tonight

Posted (March 29th, 2007 at 11:57 am PST) by Josh

Due to some backend maintenance we will be performing, any actions you take on our web panel tonight (Friday, March 30th, 2007) between 3am and 6am PDT will not actually take effect until 6am.

Again, any changes will be queued and, starting at 6am, processed… they’re just not going to happen in the normal 5-10 minutes or so!


5:33 pm

Major Network Outage.. (resolved)

Posted (March 28th, 2007 at 5:33 pm PST) by Josh

We just had a drop in our network for about half our servers.. if you can’t access your site or email now, please hold on.. we are looking into it and will update this page as soon as we have more details..

7:15PM PST

We’re still actively trying to diagnose and fix the problem.. it seems to possibly be a physical problem with one of the fiber cables between two of our data centers. We’ll post further updates here as we progress. Thank you very much for your patience and we apologize for the downtime this is causing.

7:20PM PST

Scratch that.. it turned out to be a failure in one of our 10 gigabit network interfaces. We’ve swapped it out for a spare one and thing seem to be working now.. there may be some residual problems where some servers need to be rebooted, however the root cause of the problem is fixed. Please let us know if you are still experiencing any downtime.
Our apologies again..


1:55 pm

File Server Head Unit Swap (resolved)

Posted (March 28th, 2007 at 1:55 pm PST) by Josh

We will be swapping the head unit for the file server that seems to be causing all the slow site and downtime problems affecting some people with the mail server “spacey” (you can see if thats from https://panel.dreamhost.com/ by clicking the Account Status link in the top-right).. there should be one or possibly two brief (under ten minute) periods of downtime for websites (not email) starting right now.. we’ll let you know when this is complete and how it went soon!

Okay, all done! It seems to have worked.. we actually switched the head unit for one twice as powerful, and now will be able to offload data much more quickly.

Our sincerest apologies this took so long to get resolved!


6:05 am

Power Strip Replacement

Posted (March 28th, 2007 at 6:05 am PST) by Patrick

Two power strips have lost their management interface and need to be replaced. They will be swapped out Thursday night starting at 11 p.m. PST. The first power strip that will be switched out will affect the following servers:

andor, beethoven, cyrax, dimarco, postac, thorin, and zebes

The second power strip replacement will follow shortly after the first and affect these servers:

avalon, boyd, burbank, cognac, jake, jawbreaker, sawtelle, and yoda

The downtime window for each swap is an hour, although actual downtime should be less than that.


3:44 pm

Reminder: Central database maintenance tonight

Posted (March 27th, 2007 at 3:44 pm PST) by DreamHost

Remember, we’ll be doing central database maintenance tonight beginning -Tuesday-, March 27 2007 at 22:00:00 PDT. Downtime window is 6 hours, expected downtime within that window is 4 hours. This will take the following services offline:

DreamHost Webpanel
DreamHost-provided Webmail

All other mail services will remain online, including pop3, imap, and smtp, only the DreamHost webmail install will be offline. Customer sites and databases will also function as normal.

EDIT: The day was previously listed as Monday, which is of course last night.. The date was correct, and it is indeed tonight. Sorry for the confusion.

Update: All services back online as of 11:54PM.


 
© 1996-2007, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
10:07 am

Nehi bad hardware

Posted (March 27th, 2007 at 10:07 am PST) by Terri

We’ve been investigating the cause of nehi’s continued crashing, and have discovered that the hardware is faulty. Therefore we will begin moving everyone over to new hardware in a few minutes. This process should only take about two hours, at most, to complete, and there will be minimal downtime – much less, in fact, than what nehi users have been experiencing!

Thank you for your patience and we apologize for the inconvenience.

Update 11:41 A.M. PDT: Nehi is on new hardware, and sites are starting to come back up as configurations and DNS catches up. Nehi users with any further issues can contact support at support@dreamhost.com if they still have lingering issues.


2:41 pm

Site Slowness (resolved)

Posted (March 26th, 2007 at 2:41 pm PST) by micah

The NFS issues we posted about on Friday are still affecting many of you. We are working as quickly as possible to offload the overworked filers but unfortunately it is a time consuming process. The same servers are likely to be affected:

7up arizona boba calpico cappuccino caprisun chai clamato coffee coke crush crystallight culligan dietrite eggnog espresso flipside geyser gingerale hansens hi-c hiball horchata izze jolt martinellis milk monster mrpibb mug nehi nesbitt nestea ninja optimizer orangebang ovaltine pellegrino perrier punch refresco riptide rootbeer schweppes shasta silk smoothie snapple squirt stewarts sunkist tab tampico tang tea tizer tropicana yoohoo

We’re very sorry for the poor site performance this is causing and hope to have it resolved shortly.

Update: I apologize that we haven’t posted this sooner – it’s taking longer to resolve than we had hoped initially due to the fact that the filer involved is pegged at 100% utilization. We are offloading data but it’s going very slowly (our current estimate is that it will finish by this weekend) because of the volume of data and the state of the server. Our only option to speed it up would be to remove the other causes of load on that machine (meaning customer servers that are accessing data from there – we’d really rather not turn off the hosting machines connected to that filer as that would really have a negative impact on a lot of people so we have elected to let it run its course). I’ll make sure that if we get any more useful information we will pass this along in a more timely manner.

Update: We are trying one option to see if it speeds things up without turning off any customer servers – rebooting the filer. It should take no more than 2 minutes and we’ll be watching it very closely.

Update: the reboot didn’t do as much as we hoped – rather than wait so long for the data move to complete we are going to replace the filer itself. I do not have an ETA at this time but the admin team is starting work on this as I type so we’ll get it taken care of as quickly as possible.

RESOLVED! See http://www.dreamhoststatus.com/2007/03/28/file-server-head-unit-swap/


6:52 am

Cappuccino having hardware issues, being moved to new hardware

Posted (March 26th, 2007 at 6:52 am PST) by Justin@DH

The cappuccino server is having issues and isn’t cleanly rebooting. It is being moved to new server hardware right now. Our apologies for the downtime, the services will be coming up shortly.

— UPDATE —

The move is complete and sites are being hosted normally again. Thanks for your patience.


10:43 am

Server: shocktart

Posted (March 25th, 2007 at 10:43 am PST) by Steven

It has been found that this server is currently undergoing an attack. An admin is already working to resolve this. I am unable to offer an ETA for resolution at this time but please be sure to check back here for further updates.

Update (13:29): It looks like shocktart is in the clear now and is operating normally. We’re continuing to monitor it for any lingering issues. If you are on this server and your site is still inaccessible please contact support@dreamhost.com.


1:15 pm

NFS inflated loads (resolved)

Posted (March 23rd, 2007 at 1:15 pm PST) by brians

We are experiencing connection problems with the file storage system that the following machines are mounted to. This is causing inflated loads on some of the machines. As a result making your website unresponsive. Our Admins are diligently working towards resolving this issue. We sincerely apologize for any inconvenience this may have caused. Please check back here for future updates.

7up arizona boba calpico cappuccino caprisun chai clamato coffee coke crush crystallight culligan dietrite eggnog espresso flipside geyser gingerale hansens hi-c hiball horchata izze jolt martinellis milk monster mrpibb mug nehi nesbitt nestea ninja optimizer orangebang ovaltine pellegrino perrier punch refresco riptide rootbeer schweppes shasta silk smoothie snapple squirt stewarts sunkist tab tampico tang tea tizer tropicana yoohoo

RESOLVED! See http://www.dreamhoststatus.com/2007/03/28/file-server-head-unit-swap/


4:32 pm

Yet more DreamHost central database maintenance

Posted (March 22nd, 2007 at 4:32 pm PST) by DreamHost

We will once again be performing more maintenance on the dreamhost central database beginning Monday, March 27 2007 at 22:00:00 PDT. Downtime window is 6 hours, expected downtime within that window is 4 hours. This will take the following services offline:

DreamHost Webpanel
DreamHost-provided Webmail

All other mail services will remain online, including pop3, imap, and smtp, only the DreamHost webmail install will be offline. Customer sites and databases will also function as normal.


6:30 am

Riffraff mysql server having issues (resolved)

Posted (March 14th, 2007 at 6:30 am PST) by Justin@DH

The riffraff mysql server stopped responding suddenly, a tech is on the way to the data center to check what happened to it. Should the hardware have problems, we will move the databases to new hardware. Check back for updates.

— UPDATE —

It is rebooted and back up and running fine. We will continue to monitor it for any further issues.


3:16 pm

Moving some file servers

Posted (March 12th, 2007 at 3:16 pm PST) by DreamHost

Tomorrow night, Tuesday March 13, 2007, at 10PM Pacific time (http://www.timeanddate.com/worldclock/) we will be moving two file servers down the hall to be localized to their web/mail servers for the “swarthy” cluster. This will disrupt email, web, ftp/sftp, and shell service for anyone on the affected file servers. This it to make those servers go faster, and free up some space in our cramped primary datacenter. Downtime window is 2 hours.

To find out if your server is affected, head on over to https://panel.dreamhost.com/ , and in the upper left right, click “account status”. Your email server is also your cluster! If you are on “swarthy” you have a chance of being affected. We would have to list out affected users, not affected servers Re: comment #1, and we don’t list usernames on the blogs.

Duh, My other left.

All done!


1:32 pm

Network slowness for the past hour

Posted (March 9th, 2007 at 1:32 pm PST) by DreamHost

It appears that I, in a stroke of brilliance, mis-pasted the same IP addresses into both of our core routers. This caused an effective bridging loop as both of our routers fought for the ip address. This pegged their CPUs and the uplink between them. We have removed the offending IP addresses and the system seems to have recovered. This was a problem for approximately the previous hour and 15 minutes, ending at 1:30PM Pacific time.


10:27 am

Network upgrade!

Posted (March 9th, 2007 at 10:27 am PST) by jeremy

Update: We’re all done with the upgrade. The available bandwidth between facilities is now 3x greater than it was before, with the ability to double the current amount at a moment’s notice. We hope this takes care of all of the recent issues.

We’re upgrading the network very soon so that we can alleviate the network slowness we’ve been experiencing. We’ve already transitioned most of the customers in our second facility over to the new network hardware, but we’ll be switching over to the new BEEFY cross-connect into our main facility shortly. Downtime shouldn’t be any more than a few seconds while the link comes up, but feel free to contact support if you notice any extended outages.


 
© 1996-2007, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
7:49 am

Server “Fidget” is having hardware problems. (Updated)

Posted (March 9th, 2007 at 7:49 am PST) by David

Unfortunately hardware problems have brought fidget to its knees. Our admins are in the process of replacing it at this moment. It should be back on-line again in about 15 minutes. Another update will be posted once the work is complete.

I apologize for the inconvenience.

Update: Fidget now is running on new hardware and seems to be doing fine. Unfortunately it took a bit more than 15 minutes to do, but it’s done. If you have any problems with services on this server please submit a support request for assistance.


2:23 pm

More DreamHost central database maintenance

Posted (March 8th, 2007 at 2:23 pm PST) by DreamHost

We will be performing more maintenance on the dreamhost central database beginning Monday, March 12 2007 at 22:00:00 PDT. Downtime window is 6 hours, expected downtime within that window is 4 hours. This will take the following services offline:

DreamHost Webpanel
DreamHost-provided Webmail

All other mail services will remain online, including pop3, imap, and smtp, only the DreamHost webmail install will be offline. Customer sites and databases will also function as normal.


10:29 am

Gradius web server having networking issues

Posted (March 8th, 2007 at 10:29 am PST) by Justin@DH

The gradius web server is having trouble with the public network interface, an admin is checking it out right now at the data center. If it is found to be a web server issue and not networking hardware, the server hardware will be replaced. Please check back for updates.

— UPDATE 10:43:57 PST —

The admin was able to fix the issue and the server is back up and running now. If it reoccurs we will have to suspect the machine and therefore move it to new hardware. Sorry again for the downtime and inconvenience.


7:40 pm

Panel Login

Posted (March 7th, 2007 at 7:40 pm PST) by DreamHost

If you have trouble logging into the panel clear all of your panel.dreamhost.com and dreamhost.com cookies and try again. The table Lock today was very long due to the size of the session table and we cleared it out in order to bring the panel back online faster. In the end it was decided having to re login to the panel was better then having the panel down while old (mostly abandoned) session table data was loaded.


4:24 pm

Wiggle Moving to New Hardware

Posted (March 7th, 2007 at 4:24 pm PST) by glen

The web server Wiggle is being failed over to new hardware . Total downtime will be ~1 hour . We will update as the move finishes.

Update: Failover completed. If you have any further issues with your site on Wiggle, please write in to support.


9:26 pm

Reminder: DreamHost central database maintenance

Posted (March 6th, 2007 at 9:26 pm PST) by DreamHost

Just a reminder that the scheduled maintenance of the DreamHost central database will begin in approximately 35 minutes, at Tue Mar 6 22:00:00 PST 2007. This will take the DreamHost Webpanel offline for up to 4 hours.

02:25 PST, 10:25 GMT – The Database Maintenance is progressing. There are a couple new indexes taking longer then expected to create and the current ETA for completion is 03:30 PST. Apologies to the Europeans currently at work!

02:45 PST, 10:45 GMT – Webmail is back online.

03:20 PST, 11:20 GMT – Starting to bring central web services back online.

03:30 PST, 11:30 GMT – All services online.

18:23 PST – Planned “Online” maintenance that has been ongoing since the offline maintenance last night caused a table lock that made the panel unresponsive. Will be back online shortly. Sorry for the interruption.


3:35 pm

Sporadic brief network outages.

Posted (March 6th, 2007 at 3:35 pm PST) by jeremy

Updated: We believe we’ve figured out the cause of these brief outages and are implementing a permanent fix right now. We’ll be keeping a close eye on the issues, but we’re fairly certain that we’ve worked out the issue.

We’ve encountered a few brief network outages today (lasting 1-5 minutes each). We have a few leads and are currently looking into the problem. We’ll post more information as it becomes available.

Happy DreamHost Networking Team!


3:17 pm

One-Click Installs: WordPress 2.1.2 released, critical update

Posted (March 2nd, 2007 at 3:17 pm PST) by DreamHost

A pretty bad security exploit made it into WordPress version 2.1.1, so if you upgraded to that within the last couple of days, make sure to upgrade to 2.1.2 as soon as possible. More information is available from the WP staff:

WordPress 2.1.1 dangerous, upgrade to 2.1.2

If you are unable to upgrade your install of WordPress 2.1.1 at this time, you can also just set up add a .htaccess rule to block access to “theme.php” and “feed.php”, with a rule something like this:

<Files theme.php>
order allow,deny
deny from all
</Files>

<Files feed.php>
order allow,deny
deny from all
</Files>

Update at 5:54pm pst by JamesH; The rush of our vigilant and security conscious customers to upgrade their blogs has put something of a strain on our one-click installer, and so there’s currently a backlog of one-click installs and upgrades. Right now we’re estimating it should be caught up within three hours.


9:39 am

File server Issues

Posted (March 1st, 2007 at 9:39 am PST) by brians

We are experiencing connection issues with file storage server that’s causing inflated loads on the following machines. This will explain any delay you’re encountering when attempting to view your website. As well as any problems that you are encountering while trying to access your e-mail. Our Admins are currently working to resolve the issue. Again, we apologize for any inconvenience this may have caused and appreciate your continued patience. Please check back for future updates.

7up amp aquafina bawls brisk chai clamato coffee coke culligan dads dasani endless evian fetish1 fiji flipside fresca fuze gatorade geyser gingerale hansens hi-c horchata ibc inko izze koolaid lipton milk mountaindew nesbitt nestea niagara ninja odwalla oj pellegrino pepsi perrier picndel propel right riptide rootbeer seltzer shasta smoothie sobe sparkletts squirt stewarts tab tang tea tonic tropicana voss welchs xeko spaceymail-a1 spaceymail-a2

Update (March 3, 2007, 6:35 PM PST)

This issue is resolved. If you’re still having any problems with any of the servers listed above please contact support.

UPDATE ( Mar 2, 2007 @ 4:57 PM PST )

The update is about 93% complete. Shoule be complete in a few hours. Please check back for future updates.

UPDATE(Mar 2, 2007 @ 10:38 AM PST)

The fileserver is still updating. It’s currently 81% complete. Again, we truly appreciate your continued patience. Please check back for furtue updates.

– UPDATE: 2:32 PM PST –

The network fileserver is still updating. It’s about half way done. Please allow a couple more hours for this processs to complete. Some machines may still experience a high load during this time. Again, we apologize for the inconvenience this may have caused. Please check back for future updates.


10:00 am

One-Click Upgrades: WordPress 2.1.1, WP-Cache 2.1.0, MediaWiki 1.9.3

Posted (February 28th, 2007 at 10:00 am PST) by DreamHost

WordPress 2.1.1 is a “low-to-medium” priority update with a few bugfixes. WP-Cache has been needing an update for a while, sorry for the delay! WP-Cache is installed but not enabled by default — you’ll have to log in to your wp-admin page and go under the “Plugins” page to activate it.

WordPress 2.1.1 changelog
WP-Cache 2.1.0 changelog

MediaWiki 1.9.3 fixes a possibly AJAX security exploit in IE7.

MediaWiki 1.9.3 changelog

Upgrading your One-Click Installs can be done by logging in to panel.dreamhost.com, clicking on the “Goodies” tab and then on the “One-Click Installs” page. On that page, upgrade links should appear next to each of your upgradable applications.


 
© 1996-2007, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
5:19 pm

DreamHost central database maintenance

Posted (February 27th, 2007 at 5:19 pm PST) by DreamHost

We will be performing maintenance on our central database beginning next Tuesday night at 10PM PST (Tue Mar 6 22:00:00 PST 2007). The downtime window for this maintenance is 4 hours, expected downtime within that window is 2 hours.

This will take the DreamHost Webpanel offline entirely. All other services will remain up as normal.


10:40 am

Snoopy mysql server issues

Posted (February 26th, 2007 at 10:40 am PST) by Justin@DH

The snoopy mysql server is having issues at this time, responding slowly over the network. It is being worked on now and should be working normally again shortly. You can check which mysql server you are on in the control panel under Status -> Mysql Usage. Our apologies for the issues.

– UPDATE: 12:11 PM PST –

The server seems to have some hardware issues, the data is being moved to a new faster mysql server right now. As soon as the data copy and configurations are complete, the databases will be working again. Again, sorry for the downtime.

– UPDATE: 2:32 PM PST –

Snoopy’s databases have all been successfully moved to the new server, einstein. The final moves were completed around 2PM.


9:59 am

Residual network issues still

Posted (February 26th, 2007 at 9:59 am PST) by andrea

Unfortunately, we’re still experiencing network problems due to the outage yesterday (see posts below). It consists of random network drop outs that is affecting various services, including the web, mail and mysql. The admins are working on containing it, and we’ll be posting updates as soon as we know further. We apologize for all inconvenience this may be causing, and appreciate your continued patience and understanding.

Updated: After monitoring the network for the past 24 hours, we believe that all of the previous issues have been resolved. If you believe this to not be the case, please contact support@dreamhost.com with as many details as you can.


8:36 am

Mail delay on junk filtered domains

Posted (February 26th, 2007 at 8:36 am PST) by andrea

On our system, mail to filtered domains is coming in through the junk mail machines, which then distribute processed mail to the regular mail machines. As mail service was affected by the outage yesterday, the junk machines got backed up, and are still. We have six of them, so mail is being processed as it comes in, but it’s slow. No mail is lost.

We appreciate your continued patience and understanding. Please check back here for updates.


2:56 am

Power Outage Update

Posted (February 25th, 2007 at 2:56 am PST) by Justin@DH

Power has been restored to the building and all servers are back up and running. There currently is an issue with a blade in one of the core routers and it is being replaced, so the servers are still unreachable until this issue is corrected. We are working to get service restored as soon as possible. Our apologies for the inconvenience.

– UPDATE 04:51 PST –

The supervisor in one of our core routers went kaput when the power went off. It was replaced shortly after the original posting. Further problems arose with corrupt configurations left over from the old supervisor. These have been restored and we have re-established connection to seven out of our eight uplinks. We are currently working on the eighth uplink and everything is coming back to normal operation now. We are close to being 100% and thank everyone for their patience. As always, check this space for updates.

– UPDATE 11:21 PST –

The router is up and healthy, and the majority of our network is back up and running. However, a few of our servers are having issues because their file servers are not talking to the rest of the network correctly. There is a link down from one part of the datacenter back to the routers and that is causing the file server issues. If you are on a server whose name is a beverage, this will effect you. We are working very hard to resolve this issue, and we apologize for the inconvenience.


7:28 pm

Planned Power Outage

Posted (February 22nd, 2007 at 7:28 pm PST) by micah

We were just informed one hour ago from our building management that an emergency maintenance window is scheduled for this Saturday morning, February 24th, at 1AM PST. UPDATE! The building has just changed the window to be 23 hours later.. it is now scheduled for Sunday morning, February 25th, at 12:01AM PST. This maintenance window involves taking our entire building’s power offline for approximately 3 hours. The word from the building:

It was discovered by ABM Engineering during the power monitoring
equipment installation, that a Phase C Conductor Cable on UPS #4
has been compromised to the point that immediate action to
repair it is necessary or a ground fault will occur to the building
systems.

Since only one of our three data centers is losing power, not all servers will be shut down. Unfortunately, our core routers and our upstream providers are in that data center, so there will be no network to any servers during the window. Gaah.

Since we have advanced (albeit limited) warning of this event, we will be onhand to physically power off all of our equipment at 11:15PM PST. Barring any unforeseen issues (on our end or the building engineer’s) we plan on having everything back up by 4AM PST, hopefully much sooner.

This outage will affect everyone’s sites and email. Email sent to our customers during this outage will be deferred on the sending server and will be delivered after service is restored.

We apologize for the inconvenience this no doubt causes, as well as for the short notice. We will be posting updates as possible during the outage.


1:08 am

Dreamhost.com DNS zone troubles

Posted (February 22nd, 2007 at 1:08 am PST) by DreamHost

The dreamhost.com DNS zone is currently having troubles, so most hostnames within this zone are not resolving. This shouldn’t have any effect on customer sites’ DNS, only central DreamHost services that use the dreamhost.com domain (wiki, panel, etc).

We’re looking into the cause and getting things fixed up.

Update Thu Feb 22 03:18:44 PST 2007: The authoritative DNS servers are returning records correctly again, and things are returning to normal as cached answers expire.


1:37 pm

PHP 5.2.1 Upgrades under way

Posted (February 21st, 2007 at 1:37 pm PST) by Dallas

We missed our target date by a couple of days but that has allowed us to jump to php 5.2.1 instead of 5.2.0. We will be installing the new packages little by little on all of our servers beginning today. If you are using PHP 5 (the default is still php4 so you would have had to picked it when you set up the domain), and you have an issue you believe may be related to this upgade, let our support team know.

As previously referenced on the original status post…

http://www.dreamhoststatus.com/2007/02/08/php5-downgrade-completeupgrade-scheduled-for-0219/

here is a list of known affected applications…

– WordPress versions under 2.1
Any version of wordpress under 2.1 will have problems after the upgrade to php 5.2 , the latest version can be downloaded at:

http://wordpress.org/latest.zip

Information on how to upgrade can be found here:

http://codex.wordpress.org/Upgrading_WordPress

– MediaWiki versions under 1.9.2
Any version of MediaWiki under version 1.9.2 will have problems with PHP 5.2. For information on upgrading please see http://www.mediawiki.org.

– Drupal
For detailed information on how to fix and upgrade Drupal please see http://drupal.org/node/102114 and http://drupal.org/node/92802

– Zencart versions under 1.3.7
Any version under 1.3.7 will require an upgrade. Info on upgrades can be found here http://www.zencart.com/

Another work-around is to change your site to use PHP version 4 in the Dreamhost Panel, but it is still generally best to upgrade your software to avoid security risks.


1:53 pm

Mail checking issues — Fixed

Posted (February 20th, 2007 at 1:53 pm PST) by andrea

Connection to one of our file storage servers is causing inability to check mail for some of our customer (only one group of servers -cluster- is affected). Example: ERROR: Connection dropped by IMAP server in webmail. We’re working on resolving the problem, and we apologize for the inconvenience.

Please check back for updates.

Update at 1:00pm Pacific – mail is fixed.


6:48 pm

Rootbeer fizzled out!

Posted (February 19th, 2007 at 6:48 pm PST) by Brian@dreamhost

Our server Rootbeer crashed a little bit ago. While coming back up, I saw a kernal panic, so I decided to move it to new hardware! Everything is reconfiguring to the new machine, and should be up soon!

Sorry about that!

Happy Sarsaparilla-fixit-team! =)


 
© 1996-2007, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
2:36 am

File server issues (Finished!)

Posted (February 19th, 2007 at 2:36 am PST) by kitchen

One of the fileservers that smithers, ralphie and homer mount is having some issues and some sites are being affected. Our fileserver guru is loooking into right now and hopefully will have a resolution soon. Check back here for updates on the progress.

We sincerely apologize for the inconvenience.

UPDATE(Feb 19 @ 8:06AM Pacific): It appears that a file server lost a volume sometime last night. The file server suffered a multidrive failure. This caused the raid5 volume to fail. We have replaced the drives and are restoring from backups. This will take at least an hour or more to happen. Customer sites who are affected will come online as the data is restored. There are 36 users directly affected by this issue. This is the list of servers affected, since we do not post usernames here:

homer otto dawber gotcha tak homer marge lisa chalmers wiggum nelson kearny cletus ralphie flanders burns mcclure seamus millhouse smithers willie annie cheeky dib

We’re sorry about the previous sparse information.

UPDATE(Feb 20, 2007 @ 10:17AM Pacific): The file server has been back online since the previous posting. The only people affected by this are the 36 people who were missing the data in their home directory. We’re restoring it as fast as we can, but 4TB of data simply takes a long time to copy. Only 18 users left to copy. The servers above should not be feeling any ill effects of not having this data there. We’re sorry about the downtime. The data is copying as fast as the hard drives will let it.

UPDATE(Feb 21, 2007 @ 10:55AM Pacific): The restore program is on to the last 2 users. We are also working on ways internally to make this go faster. This is the first time we’ve had to run a restore of this magnitude on this particular type of file server. Hopefully, if/when there is a next time, we will have a faster method in place. The backups are all there, it just takes a very long time to copy them over.

UPDATE(Fed 22, 2007, @ 10:34AM Pacific): The restore program is finished! If you are still missing files from this server you should contact support.


4:48 pm

Gir kernel panic.

Posted (February 17th, 2007 at 4:48 pm PST) by DreamHost

It is complaining about one of the CPUs so we are working with the admin oncall to see if it’s just needing a new kernel or to be rebuilt on new hardware. We will post once this is resolved (it should take no more than 30 minutes). Once this is fixed gir should be quite stable again.

Update: gir has been failed over to new hardware and is doing well – we do apologize for the inconvenience!


9:46 pm

File Server RAM Upgrade!

Posted (February 15th, 2007 at 9:46 pm PST) by Patrick

One of our file servers is in desparate need of extra RAM. It will be going down at 10:30 for an upgrade. This will not affect most customers. It may however affect some users (not all) on the following servers:

absinthe admiralty alameda alondra altoid archibald arrow atlantic avalon aviation babyruth badillo barranca barry bazooka beaudry beverly bourbon boyd broadway bundy burton butler caramello catalina centinela century cheeto cheezit chex cordial crenshaw cupcake dc-asparagus dh-spunky-mail dingdong doheny dorito dumdum edendale euclid federal firestone fletcher foothill formosa fundip garland garth garvey goober gower grand gummi haven heath highland hillhurst hillmont holt hoover hope hops hyperion jalapa jawbreaker jerky juniormint jurupa kiki kitkat lankershim lemonhead licorice lifesaver limbo-spunky locust look lucas manchester mariposa melrose mills moorpark nash nerds ninja nougat ocean olympic ord overland oxnard paramount payday pez pico planters plenty pocky popcorn powerbar pretzel price razzle redhot redondo reeses rexford ritz rockyroad rodeo rollo rosecrans rossmore rum runts sake saltine selma sepulveda shocktart skybar slauson smores snicker snocap sourpatch spree starburst sugardaddy sunset taffy temple toffee tootsie torrance twinkie twizzler union vermouth vine virgil warhead western whittier wilmington xenotime

Estimated down time for the file server is right at 20 minutes. Check this space for updates.

– UPDATE Thu Feb 15 22:38:22 PST 2007-

The file server is back up with more RAM and everything is mounting it properly!


5:41 pm

Ruble server getting new hardware

Posted (February 14th, 2007 at 5:41 pm PST) by kitchen

Ruble’s power supply decided to go on strike, so we’re moving it over to new hardware. This process shouldn’t take longer than half an hour. Check back here if you’re still having problems, and don’t hesitate to contact support if after the move you’re still having problems accessing your domains.

Very sorry about the inconvenience.

This has now been completed. Contact support through the control panel to let us know if you have any problems with your files on ruble.


3:19 pm

File server battery swaps! (Done!)

Posted (February 13th, 2007 at 3:19 pm PST) by DreamHost

I need to swap out batteries in three of our file servers. This is to help them maintain the integrity of YOUR data in the event of a power outage, or other hardware failure. These generally take about 3-4 minutes each, during which time none of your data will be lost, but you will also not have access to it. I’m sorry for the short notice, but our file servers also give us very short notice as well!

This will affect the Spacey, Swarthy, and Spunky clusters.

First one complete (spacey): Tue Feb 13 22:29:01 GMT [rc:notice]: The system was down for 183 seconds

Second one complete (swarthy): Tue Feb 13 22:40:12 GMT [rc:notice]: The system was down for 167 seconds

Third and last one complete (spunky): Tue Feb 13 22:46:27 GMT [rc:notice]: The system was down for 156 seconds


6:01 pm

Mail authentication outage

Posted (February 12th, 2007 at 6:01 pm PST) by DreamHost

We just experienced an authentication outage for all of our mail clusters. This has been resolved.

A network hiccup caused our load balancers to reset all connections to the backend authentication databases, which in turn caused all mail servers to need a reset to get their connections back.


11:17 am

Larchmont getting failed over

Posted (February 9th, 2007 at 11:17 am PST) by chih

Larchmont is unable to mount any remote filesystems and powercycling the server did not help. To get the server back up and running as quickly as possible we are failing over the server. The server should be back up in about an hour. Sorry for any inconvenience this has caused you.

UPDATE: The move went well and everything is back up and running. Thank you all for your patience!


9:34 am

Niagara not flowing

Posted (February 9th, 2007 at 9:34 am PST) by Terri

Niagara has a busted NIC, so we’re moving everyone over to new hardware. This should take no longer than two hours, and we’ll be updating this post once it’s up and running again. My apologies for any inconvenience this has caused.

Update: Niagara is back in business. If you are on Niagara and are still experiencing downtime related to the move, please contact support.


5:19 pm

One-Click Install Update: MediaWiki to 1.9.2

Posted (February 8th, 2007 at 5:19 pm PST) by DreamHost

As always with upgrading MediaWiki, after the upgrade make sure to run those database upgrade scripts located in the /maintenence/ subfolder of wherever you installed Mediawiki. More specific instructions on how to do this are contained in the post-upgrade e-mail sent by our One-Click Installer Robot.

Mediawiki 1.9.2 changelog

To upgrade your One-Click Install of Mediawiki, log in to panel.dreamhost.com, click on the “Goodies” tab, then on the “One-Click Installs” link, and then click on the [upgrade] link next to your install of MediaWiki.


4:33 pm

PHP5 Downgrade Complete/Upgrade Scheduled for 02/19

Posted (February 8th, 2007 at 4:33 pm PST) by jordan

The downgrade has been completed so all sites should be functioning normally at this time. We have re-scheduled the php5 upgrade for February 19th. Here is a list of software that will be affected by this upgrade and steps for you to take to insure that your site is not affected by this upgrade. Note: Any software that was installed via our one click feature in the panel can also be upgraded by going to the One Click Section in the Goodies tab of your panel https://panel.dreamhost.com/index.cgi?tree=goodies.installer&

– WordPress versions under 2.1
Any version of wordpress under 2.1 will have problems after the upgrade to php 5.2 , the latest version can be downloaded at:
http://wordpress.org/latest.zip
Information on how to upgrade can be found here:
http://codex.wordpress.org/Upgrading_WordPress

Another fix is to change your site to use PHP version 4 in the Dreamhost Panel, but it is generally best to upgrade to avoid security risks.

– MediaWiki versions under 1.9.2
Any version of MediaWiki under version 1.9.2 will have problems with PHP 5.2. For information on upgrading please see http://www.mediawiki.org.

– Drupal
For detailed information on how to fix and upgrade Drupal please see http://drupal.org/node/102114 and http://drupal.org/node/92802

– Zencart versions under 1.3.7
Any version under 1.3.7 will require an upgrade. Info on upgrades can be found here http://www.zencart.com/

We apologize for the inconvenience.

UPDATE 02/15 —
The update previously scheduled for monday 2/19 is being moved to the 20th due to the holiday.


 
© 1996-2007, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
8:53 am

PHP5 Issues

Posted (February 8th, 2007 at 8:53 am PST) by Terri

Many of you have noticed that in the last 24 hours that your Mediawiki, WordPress, or Zen Cart install was not working inexplicably. Customers may have found either internal server errors or completely blank pages. This is due to a problem we had with a PHP5/Zend Optimizer upgrade. Although we had tested the upgrade, and had upgraded to a small set of servers first, a problem occurred and it came to our attention that the upgrade had upset the operations of these applications.

We are in the process of downgrading PHP5 and Zend Optimizer right now. The majority of our servers are now successfully downgraded, but it will be about 20 minutes to a half an hour before everyone is completely fixed. We will be investigating the cause of the errors, as well as the events that led up to the system-wide problem so that this will never happen again.

We apologize for any inconvenience the upgrade may have caused. Please contact support if your site is affected and is not up and running due to this issue within 30 minutes. We thank you for your patience.

Update: The downgrade is now complete on all of our servers. Please contact support for any further problems regarding this issue.

UPDATE 2pm: We are currently running another downgrade on all our shared web servers as the morning one wasn’t complete. You may experience similar errors as before that downgrade, sorry about that. The process should complete within a few hours. We appreciate your continued understanding and patience.


3:13 am

Go will not go!

Posted (February 8th, 2007 at 3:13 am PST) by Terri

My apologies, residents of go, but there is a kernel panic on the server that has happened twice already. I’m moving everyone over to new hardware, and this should take no longer than an hour or two. I’m very sorry for the inconvenience.

Update: Go is up and running and all services seem to be running optimally. Please contact support if you have any problems with the go server.


9:19 am

Baht server getting new hardware — completed

Posted (February 7th, 2007 at 9:19 am PST) by andrea

A little bit ago baht started reporting very high load and we noticed disk errors in the logs. Soft and hard reboot didn’t help, so we’re moving the server to new hardware. This process shouldn’t take longer than half an hour. Check back here if you’re still having problems, and don’t hesitate to contact support if after the move you’re still having problems accessing your domains.

Very sorry about the inconvenience.

This has now been completed. Contact support through the control panel to let us know if you have any problems with your files on baht.


5:03 pm

More panel database maintenence

Posted (February 5th, 2007 at 5:03 pm PST) by DreamHost

Sorry for the short notice once again. Tonight, Monday, Feb 5 22:00:00 PST 2007 (this coming Monday at 10PM) we will be performing some more central database maintenence. Downtime window extends til 23:59:00 PST, but exepected downtime in this window is 60 minutes. This downtime will affect the following services:

DreamHost Webpanel
DreamHost wiki

All customer sites, customer databases and email will function as normal.


10:24 am

Mail delay on junk filtered domains – This is now fixed

Posted (February 5th, 2007 at 10:24 am PST) by andrea

Two out of our nine junk filter machines are having a hard time processing mail this morning. This means that incoming mail is going through but getting delayed, if they happen to be handled by those two machines. This only affects domains/addresses that have junk mail filters enabled. No mail is lost.

The admins have been notified and are looking into it. Check back often for status updates.

We apologize for the inconvenience.

Update at 11am PST: this is fixed and mail has caught up.


8:05 pm

Cyrax

Posted (February 2nd, 2007 at 8:05 pm PST) by Brian@dreamhost

Cyrax lost it’s network card, so we’re moving it over to new hardware! We should have everything restored shortly. Sorry about the downtime.


3:14 pm

Adams failed.

Posted (February 2nd, 2007 at 3:14 pm PST) by mir

We are moving it over to new hardware and this should be done in the next ten minutes or so (with an estimated maximum of 30 minutes from now). We apologize for the inconvenience.


10:48 pm

Dedicated Server Moves

Posted (January 31st, 2007 at 10:48 pm PST) by Patrick

As a reminder to the dedicated customers e-mailed last week, your servers are being moved now to the LAX datacenter. Estimated downtime is right around two hours.

Happy DreamHost Mule-Pack Team


5:43 pm

Emergency panel downtime tonight (done)

Posted (January 29th, 2007 at 5:43 pm PST) by DreamHost

Sorry for the short notice, but we will be taking the DreamHost Web Panel down tonight at 10PM PST (Mon Jan 29 22:00:00 PST 2007), for some emergency database maintenence. Downtime window is 2 hours, expected downtime within that window is 1 hour.

This will have no effect on customer sites, databases and email, all those services will remain online.

Update Mon Jan 29 23:18:42 PST 2007: The maintenence went smoothly, and all services are back online.


10:25 pm

Database server problems, riffraff (Update on backups)

Posted (January 25th, 2007 at 10:25 pm PST) by DreamHost

The database server “riffraff” is having hardware problems. We’re working to get them figured out and resolved and get it back online as quickly as we can.

Watch this space for updates.

Edit 10:15pm pst by Jamesh: I’ve just finished putting a new raid card in Riffraff and it’s up and working now, if you’re having any problems with your databases please contact support.

Edit 10:15am pst by Kelly: It seems there was some data corruption on the live disks due to the funky raid card. I moved the backup drive out of the old server and into a spare machine. The backups are copying over and should be available shortly via the web panel for restores! The data has been moving for about 20 minutes now, it should finish in about 15.


 
© 1996-2007, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
4:36 pm

Hooray for new Rails!

Posted (January 25th, 2007 at 4:36 pm PST) by Terri

The packages are done, testing is complete, and I’m ready to roll out the new version of Rails – 1.2.1. Slowly, of course!

I will start slowly upgrading servers with Rails 1.2.1 starting Monday, January 29. There is one caveat to this.

If you are running Typo you have to freeze your Rails! If you’re afraid your Rails application will break with the new upgrade, and you’re not planning on tweaking it to use the new version candy, freeze your Rails. I tested Rails 1.2.1 and it breaks Typo. The developers recognize this and recommend freezing Rails (although they’ll be making adjustments very soon). Freeze your Rails! *GONG!* Freeze your Rails! *GONG!*

You can easily freeze your Rails by using SSH to login into your account. Go into the directory that has your Rails application (ie cd myrailsapp.com/). This is the directory that holds public, config, lib, etc. Then, run this command:

rake rails:freeze:gems

Rake will tell you what version it’s freezing your application to, and where. After that, you’ll be safe!


3:13 pm

Ruby Gem Updates and Additions

Posted (January 24th, 2007 at 3:13 pm PST) by Dallas

We are starting to roll out some Ruby gem updates and additions. We always try to leave at least two versions of every gem installed so these upgrades add a newer version while the previous most recent version is also left installed. Let our support team know if you experience any issues with these updated gems. They are not yet rolled out to all servers so before contacting our support team, be sure to make sure your issue is related to a gem version change.

These Ruby gems are being newly installed: hoe 1.1.7, camping 1.5, unicode 0.1, ajax_scaffold 2.2.1, gruff 0.2.8

These Ruby gems are being upgraded: ParseTree 1.6.3, rubyforge 0.4.0, RubyInline 3.6.2, acts_as_taggable 2.0.2, capistrano 1.3.1, daemons 1.0.3, ferret 0.10.13, gen 0.41.0, glue 0.41.0, geoip 0.3.0, nitro 0.41.0, og 0.41.0, daemons 0.4.2


2:14 pm

Server Relocation Bonanza!

Posted (January 24th, 2007 at 2:14 pm PST) by Patrick

The server moves are almost all done! We do have a few major server moves left though. One of those will start tonight at 10 p.m. PST. The following servers will be switching racks tonight:

lenny, mcclure, millhouse, willie, seamus, burns, brandine, kearny, cletus, apu, lisa, skinner, chalmers, wiggum, limbo-postal, bigredgiant, marge, flanders, ralphie, cheeky, alexander, baht, contra, centipede, defender, joust, bob, rupee, pizarro, donkeykong, bomberman, abbazaba, smores, skor, rampart, wilmington, schnapps, mills, aero, aviation, altoid, butterfinger, kiki, hyperion, cerritos, york, poprock, hillmont, necco, shocktart, warhead, bottlecap, bazooka, archibald, taffy, limbo-spunky2, xekogame, fc23, bounty, nougat, juniormint, and reeses.

Although that looks like an enormous undertaking we, being the super smart people that we are, will break it into segments to minimize downtime! No server should experience more than 20 minutes of inaccessibility. Check this space for updates.


7:49 pm

Mailbox and email address updates backed up – Update!

Posted (January 21st, 2007 at 7:49 pm PST) by andrea

8:58PM PST
The problem is now fixed, and all updaters but one finished running. The last one is on our biggest server cluster (group), and it’s safe to say that it should finish within one hour.

Again, sorry about the inconvenience!

We’re currently experiencing problems with our automated mailbox updaters. This means that mailboxes and email addresses created or updated recently are still in their old state.

This is being looked into by our admins and we’re working on a fast resolution. We apologize for the inconvenience.

Check back for updates!


12:19 pm

Load issues on webmail machines

Posted (January 18th, 2007 at 12:19 pm PST) by andrea

We are experiencing load spikes on some of our webmail servers. We still have others to handle the webmail flow, but this problem may cause intermittent slowness and/or inaccessibility.

The admins are working on it, and it should be fixed soon. Please check back for updates.

Very sorry about the inconvenience!


8:23 pm

One-Click Install Updates: WordPress to 2.0.7, Zencart to 1.3.7

Posted (January 17th, 2007 at 8:23 pm PST) by DreamHost

Less bugs! More security! Read on:

Changelog for WordPress 2.0.7
Changelog for Zencart 1.3.7

I also fixed two bugs with our One-Click installer that have been noted for a while:

1) I added “–dereference” to all of our “chown -R” commands, which will hopefully resolve all the problems our users were having with symlinked folders having the wrong permissions on them after an install or upgrade.

2) I fixed the post-install e-mail for ActiveCollab to reflect that “/public/install” is the folder to delete, not “/install”.

As always, you can upgrade by logging in at panel.dreamhost.com, clicking on the “Goodies” tab and then on the “One-Click Installs” page, and then clicking on the “Upgrade” link next to your installation of WordPress or Zencart.


3:47 pm

Four Quick Server Moves

Posted (January 17th, 2007 at 3:47 pm PST) by Patrick

The following four servers are in the wrong place!

sunkist, punch, sprite, and shasta

They will be going down in about 10 minutes for a quick move to their proper location. This move is within the same datacenter. Total downtime per machine will be no more than 10 minutes.

– UPDATE Wed Jan 17 15:02:21 PST 2007 –

All four servers are back up and running! If you are having any problems with your server, please contact support.


12:12 pm

Central services database maintenence (Done!)

Posted (January 17th, 2007 at 12:12 pm PST) by DreamHost

Tomorrow night, starting at 10PM (Thurs Jan 18 22:00:00 PST 2007), we will be taking the following services offline for some emergency maintenence:

DreamHost Panel (panel.dreamhost.com)
DreamHost-provided webmail
DreamHost wiki (wiki.dreamhost.com)

The downtime window for this will be approximately 2 hours, expected downtime within that window is 1 hour. This will have no effect on customer sites, and customer email will continue to function normally, except for the webmail access to it.

Update: Fri Jan 19 00:45:39 PST 2007: There were some unforseen delays causing the downtime to extend about 15 minutes past the estimated window, however, all services are now back online.


2:28 pm

‘Galaga’ server having problems.

Posted (January 14th, 2007 at 2:28 pm PST) by danwy

Those of you with sites hosted on the galaga server may have noticed that your sites are unresponsive as of about 30 minutes ago. It appears that the server ran out of memory, but is not coming back up after our attempts to restart it. We are in the process of moving the server over to new hardware and hope to have it back up shortly. Sorry for any inconvenience this causes you.

UPDATE: 4:25pm (PST): After all of the configs were run on the new server, apache2 services still weren’t running. Most of the sites on the server are up and running fine, it’s just the ones that are on a apache2 services that are still experiencing problems. Our admins have been called in to look at it, and everything will be resolved ASAP. Sorry for the delay (and the false alarm).

UPDATE: 5:15pm (PST): We found the culprit and galaga should be fully back to normal (as far as we can tell). If you have any further problems with galaga, contact support.


11:47 pm

Central Database Problems

Posted (January 12th, 2007 at 11:47 pm PST) by jordan

We are currently having some problems with our central database , this will affect acess to the panel , wiki and various other internal services. Your web /mail service will not be affected by this outage. Please check back for more information as it becomes available.

UPDATE ** 1:15AM PST
Problems with the central database have been resolved.


 
© 1996-2007, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
12:15 pm

Hill users moving day!

Posted (January 12th, 2007 at 12:15 pm PST) by Terri

In about a half an hour we will be moving everyone on the server hill to the server “granola.” The reason why we are doing this is because the host name has been under attack and the only way to give everyone relief is to change their host! We apologize for this inconvenience, but the move shouldn’t take that long, an hour at most, and it should be very seamless. Nobody should notice any gaps in service.

UPDATE: We’re being a lot more careful with moving everyone than originally planned, since we thought caution being a better way to handle it. This move will be completed within two hours.


5:03 pm

Harpo Hosed

Posted (January 9th, 2007 at 5:03 pm PST) by Patrick

The web server harpo had one of it’s NICs fail earlier. It is currently being moved to new hardware and will be back up and running on a brand spanking new machine soon. Check this space for updates.


4:53 pm

WordPress One-Click Install updated to version 2.0.6

Posted (January 7th, 2007 at 4:53 pm PST) by DreamHost

After repeated death threats and a customer actually setting himself on fire in our parking lot, I’ve finally updated our One-Click Install of WordPress to version 2.0.6. It’s mainly some important security fixes, read more about it here:

Changes In WordPress 2.0.6

If you installed WordPress using our One-Click Installer, you can upgrade it by logging in at panel.dreamhost.com, clicking on the “Goodies” tab and then on the “One-Click Installs” page, and then clicking on the [Upgrade to 2.0.6] link next to your installation of WordPress.


3:47 pm

Earthlink/CBL Update

Posted (January 6th, 2007 at 3:47 pm PST) by Karl

An update on the previous CBL/Earthlink issue posted about here:

http://www.dreamhoststatus.com/2007/01/03/cblxbl-blacklist-issues-bouncing-mail/

CBL temporarily inhibited our IPs from being blocked several days ago, so our listings there (and with XBL/spamhaus) have ceased. We’re still talking with them to get this resolved permanently, and our IPs should not be blocked by them in the meantime. As many of you noticed, however, mail to Earthlink is still being blocked.

We have been emailing Earthlink since 12/30, before we ever contacted CBL. After 6 emails we unfortunately haven’t received anything but autoresponses from Earthlink’s blacklist/abuse address. We’ve spoken with their techs via chat and phone and have not been able to get through to anyone above their first tier of support; attempts to transfer to a supervisor were met with 20 minutes on hold and the call being dumped.

We will still be attempting to contact Earthlink to get this particular issue resolved, but if you are an Earthlink customer and are being impacted by this (such as if you utilize forwarding to an Earthlink address — and we know there are several thousand of you), you may wish to contact their support and let them know that we have been writing them to get information about this block and how it can be resolved. If they are so inclined, they may even contact us directly at abuse@dreamhost.com!

Finally, I do apologize for the impact on everyone. Blacklistings are no fun for anyone, and unfortunately we’ve found that large ISPs are the biggest headache to get de-listed from, as it’s incredibly hard for us to get a human response out of them. Still, we can’t fix the problem on our end (if there is one) without them responding to us to let us know what the problem is. There needs to be a reasonable amount of earnest communication on both ends for these issues to be resolved, and we are certainly trying to meet our end of the bargain there! We aren’t shy about disabling spammers when we find them, and certifiable spammers are actually pretty rare on our primary SMTP machines judging by the (few) complaints we get.

Thanks to everyone for continued patience, which understandably may be wearing thin!


1:45 pm

Server moves! (Done!)

Posted (January 4th, 2007 at 1:45 pm PST) by DreamHost

In our continued efforts to localize our server clusters physically, we will be moving the following servers to their new home (down the hall in our new datacenter!)

andor atomic chewie cyrax dali dimarco frigga fud ganymede genki gradius hug kenobi lilac limbo-swarthy medina narcissus noid postac rock samus sektor shaka slash slaw soda squirm strange thorin tron van vidar vlad wiggle windu zebes

(Update: Scratch narcissus and sektor off the list. They are the wrong formfactor for our new racks. We will be moving them to different hardware, however it will be on a future date. Your service should continue uninterupted!)

These servers, along with their mail servers, will be moved in batches. We will not bring all of the mail servers offline simultaneously, so email services should not be affected. The moves will start a little after 10pm Pacific time (-0800) Monday, Jan 8, 2007, and individual servers should be down for about 20 minutes. Check here for updates.

Update: Moved the following servers: atomic, postac, van, medina, kenobi, fud, dali, windu, cyrax, noid, shaka, lilac, frigga, strange, gradius, slaw. Due to illness with myself and my helper the rest of the moves have been post-poned until tomorrow night. We’re very sorry for the short notice.


All done! The servers have successfully migrated north up the hallway to our other datacenter.


4:29 pm

CBL/XBL Blacklist Issues / Bouncing Mail

Posted (January 3rd, 2007 at 4:29 pm PST) by Karl

We have been having issues for the past few days with mail being rejected due to several of our main IPs being listed, de-listed, and re-listed within a matter of hours in the CBL blacklist. Earthlink among many other providers use this blacklist to reject mail. The XBL blacklist also incorporates CBL into their listings.

If you have had mail rejected on an IP in the range of 208.97.132.X with a link to cbl.abuseat.org or spamhaus.org/xbl, this is the cause, and it is something that we are currently corresponding with the maintainers of the CBL to remedy as soon as possible.

From their early indications this does not appear to be due to spam complaints, but rather some misunderstandings in how our DNS and mail load-balancing works. We are hoping that we can clarify the issue and/or resolve any problems within the next couple days (if not hours). Currently we are just waiting to hear back from the CBL team again to clarify precisely what in our DNS is causing them to list us. Our overall mail DNS system has not changed since February 2006, short of the addition of new servers behind our load-balancer that all use the same pool of public IPs.

DreamHost remains committed to combatting spam, per our stringent anti-spam policy:

http://www.dreamhost.com/spam.html

You do not need to write support about this issue (or the related issue of Earthlink blocking our mail), as we are actively monitoring the latest outgoing mail logs to keep up-to-date on the situation. For original mail you are sending, you can send via Webmail (at http://webmail.yourdomain.com ) and this will go out on different IPs that should not be blocked. If you utilize email forwarding to an address that is protected by these blacklists, it is strongly suggested that (if you don’t already) you set up a mail user on our servers to also receive copies of email for your address.

I apologize for any difficulty that these blocks are causing, and appreciate everyone’s continued patience in the meantime.


3:46 pm

Relay Access Denied Errors

Posted (January 2nd, 2007 at 3:46 pm PST) by brians

I just submitted a request to have our IP removed from the Composite Blocking List. Everything should be resolved shortly. In the meantime, I can suggest that you use your ISPs smtp server to send your outgoing messages. Your email will still be from your own domain name, the only difference is in the way it is routed throughout the Internet. Instead of being sent through our servers it is processed through your ISP’s servers. Again, we sincerely apologize for any inconvenience this may have caused. Please check back here for updates.


3:18 pm

One-clicks: Joomla updated to 1.0.12, ActiveCollab updated to 0.7.1

Posted (December 29th, 2006 at 3:18 pm PST) by DreamHost

Here’s the changelogs for both of them:

Joomla 1.0.12 changelog
ActiveCollab 0.7.1 changelog

Both of these upgrades are recommended as they contain bug fixes, and for Joomla some security fixes as well. You can log in to the One-Click Install page here to do so:

https://panel.dreamhost.com/index.cgi?tree=goodies.installer&

As always, when upgrading a One-Click Install, make sure to carefully read the instructions that our One-Click Installer Robot sends you after the upgrade has completed, in case there’s any additional steps that you have to take to complete the upgrade!


2:55 pm

Rewards policy update, DreamHost trademark usage

Posted (December 29th, 2006 at 2:55 pm PST) by DreamHost

Just a heads-up for those using our popular DreamHost Rewards program!

Lately we’ve been noticing an increase in the number of people utilizing DreamHost Rewards – this is good! We’re happy that you guys are spreading the word.

However, we’ve also noticed a marked increase in people using the ‘DreamHost’ trademark in their domain names in an apparent effort to add some degree of official-ness to their referral, improve search engine rankings, etc. Whether the intent is malicious or not, this sort of thing constitutes infringement of our ‘DreamHost’ trademark and we can’t allow that to continue.

Because of that, we’ve added the following verbiage to our DreamHost Rewards terms:

Gaining referrals through misrepresentation, improper use of DreamHost intellectual property (ie. DreamHost trademarks and copyrighted material) or impersonation of DreamHost personnel is strictly prohibited and can result in loss of accrued Rewards and/or removal from the program.

In 30 days, we will prevent Rewards from accruing for any domain that contains our registered trademark (including obvious misspellings, etc). Such Rewards links will simply cease to work. We will also begin contacting the owners of those domains in order to arrange a resolution.

If you have a domain that you believe may infringe upon DreamHost trademark, please contact us directly via Tech Support as soon as possible. For those domains registered prior to today we will happily provide a 1 year domain registration once the domain is under our control so that their owners can register a different domain in its place.

If you have any questions, please let us know!

Update: Just for clarification, these changes won’t impact those using dreamhosters.com sub-domains or old-school dreamhost.com sub-domains!


4:12 am

Galaga belly-up

Posted (December 23rd, 2006 at 4:12 am PST) by Terri

Galaga’s hardware just couldn’t take it anymore. I am moving everything to new hardware. This shouldn’t take more than an hour for all services to come back online. I apologize for the inconvenience.

Update: The move went well and everything is back up and running. Thank you all for your patience!


 
© 1996-2006, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
4:02 pm

Gradius memory bad – replacing now!

Posted (December 21st, 2006 at 4:02 pm PST) by mir

It looks like the memory on the web server Gradius has gone bad – reboots are not helping and the console messages indicate that this is the problem. We are taking it down to get that replaced and will have it up as quickly as possible (hopefully no more than 15-20 minutes).

Update: the machine is back up and starting all of its services, we’ll keep an eye on it to make sure all is well now.


11:05 pm

Dedicated Server Moves

Posted (December 20th, 2006 at 11:05 pm PST) by Patrick

As a reminder to those dedicated customers that were notified last week, your servers are being moved to our new datacenter! Expected downtime is stilll right around 2 hours.

We had a network hiccup just now (12:35-12:45 Pacific time) to the datacenter these servers are in. It was just resolved, total downtime was about 10-15 minutes.


1:14 pm

Timed execution going offline for 2 hours for Looney (And Postal!) (all done!)

Posted (December 20th, 2006 at 1:14 pm PST) by DreamHost

Tonight around 6pm We will be turning off timed execution (cron) on Looney and Postal. We need to move where we store your crontab files, and the best way to do this is to simply turn off cron while we do it. 2 hours should be WAY overkill. Your crontabs will not catch up, and will either need to be run by hand or just wait for the next time!

Affected servers:

apok axl balrog blanka brazil charm curly dhalsim dozer egon guile limbo-looney moe neo ryu slappy soy tank tofu trinity vega venkman winston zangief zuul

Sorry for the late update. I completely forgot the postal crontabs were stored on the same server until I started looking at it just now.

More servers:
annie apu bart bigredgiant bitters brandine burns carl chalmers cheeky cletus dawber dib flanders gaz gimli gir glass gloop gotcha harpo homer kearny krusty larry lenny limbo-postal lisa marge mcclure millhouse monkey nelson otto pogo raiden ralphie salt seamus shemp shnooky skinner smithers tak wiggum willie yerba zim

As of about 7:15 Pacific everything should be back online relating to crontabs. Let us know if you are having any problems with them!


1:28 am

Server Moves!

Posted (December 20th, 2006 at 1:28 am PST) by Patrick

In our continued effort to localize clusters phyiscally, the following servers will be moved ‘home’ (home being down the hall).

dali, dollar, gap, io, kiki, limbo-postal, lira, medina, mounds, nutrageous, orbital, ord, rhod, rockyroad, ruble, selma, sepulveda, slash, spree, squirm, ultima, xenotime, and yerba

There will also be a few mail servers being moved from the spunky cluster. Since all of the mail servers are behind a load balancer, mail service should not be affected.

The servers will be moved in three batches starting with the mail servers, followed by two batches from the list above.
Estimated downtime for each server is right around 20 minutes. Check this space for updates!

– UPDATE Wed Dec 20 02:40:34 PST 2006 –

All servers have been moved and are back up and running! If you are having any problems at all with any of the above servers, please contact support.


11:34 pm

Last chance to donate to Santa’s Little Helpers before Christmas!

Posted (December 19th, 2006 at 11:34 pm PST) by Josh

Just a note.. we’re going to pay out (with our matching amounts!) the donations we’ve received for Santa’s Little Helpers early.. so please get any last minute donations in within the next 24 hours here:

https://panel.dreamhost.com/?tree=home.charity

Happy Holidays!


9:34 pm

Incoming Support Requests/ Replies via E-mail Delayed Since Sunday

Posted (December 18th, 2006 at 9:34 pm PST) by glen

Our support system had inadvertently held back all incoming support requests sent in via E-mail since sometime in Sunday. While we thought we had cleaned out most of the support requests by the middle of the day today, we later found out that there were 100’s waiting to still come into our support queue due to a clog in the system. It appears as if they had been building up since sometime on Sunday and we have only received them to reply to within the last hour or 2.

If you have sent in a support request / reply since Sunday and have yet to receive a response, please do not write in again. We have the request and will be answering it as efficiently as our tech support personal can handle them. This DID NOT affect support requests sent in through our control panel.

NOTE: Replying to this blog post will not get your support request read any quicker, however if it feels good DO IT!


8:19 am

Tiny Tuckered Out

Posted (December 18th, 2006 at 8:19 am PST) by Patrick

The MySQL server ‘tiny’ is suffering from disks errors causing some databases to be inaccessible. Unfortunately, due to the nature of the disk errors, we have to copy all backups from the server over to a new MySQL machine. After the copy is complete, we will then begin the restoring databases from backups. During this process, all databases on tiny will be inaccessible. Check this space for updates.

Update Mon Dec 18 11:45:05 PST 2006: All databases are back online.


6:57 pm

Networking issues resolved

Posted (December 14th, 2006 at 6:57 pm PST) by jeremy

Our network has been stable all evening, so everything should be back to normal. We’re opening a case with Cisco as it seems that the memory outage was likely triggered by a bug in the OS. We’ll update again if we get any additional information worth passing along.


2:41 pm

Networking issue update

Posted (December 14th, 2006 at 2:41 pm PST) by jordan

Our networking engineer made a change to our core2 router configuration. It caused the router to run out of ram and wipe out its configuration file. We have restored the configuration from backup and are going over it now to ensure everything is correct. You still may be experiencing site outages as we correct the problem. Please check back for updates which we will be making untill the problem is 100% fixed..


12:45 pm

We are currently experiencing networking issues.

Posted (December 14th, 2006 at 12:45 pm PST) by danwy

Some of you may have been noticing some problems with your mail and MySQL services (and possibly some others). We’re not 100% sure what is causing these network problems, or what is being affected, but our entire admin team is currently investigating the issue. Stay tuned, we’ll be updating this post once we have any new information.


 
© 1996-2006, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
10:58 pm

Dedicated Server Moves!

Posted (December 13th, 2006 at 10:58 pm PST) by Patrick

Just a quick reminder to those dedicated customers who were notified last week. Your servers will be going down at roughly 10:30 pm PST. The entire move process will take approximately 2 hours. Check this space for updates.

Happy DreamHost Dedicated Pack Mule Team!


12:15 am

Smithers being moved onto new hardware.

Posted (December 13th, 2006 at 12:15 am PST) by DreamHost

Smithers seems to have some bad memory, so we’re moving it onto new hardware right now. The whole process should take less than an hour, and leave all of our customers hosted on that machine with a shiny new webserver.


8:22 am

Sanspoof database server down (and back up)

Posted (December 10th, 2006 at 8:22 am PST) by jeremy

If you’ve noticed your databases being down, it’s likely because one of our database machines, sanspoof, simultaneously lost 2 drives shortly before 5am this morning. We have it on new hardware and are restoring all of your databases from backups as we speak. We expect everything to be back up sometime around 9am PST.

Update: Everything was back up by approximately 10am PST on Sunday the 10th. Only about 2-3 hours of data was lost on average.

Happy DreamHost Why-Must-It-Be-2-drives-At-5am Team!


4:02 pm

Upgrading memory for Zorg, Napoleon and Grappa

Posted (December 8th, 2006 at 4:02 pm PST) by mir

That should do it for upgrades today, we should see much better stability for all of the machines upgraded today as based on crashlogs they were swapping. Zorg is already back up actually, the other two should have minimal downtime.

– UPDATE Fri Dec 8 15:13:34 PST 2006 –

All three servers are alive and kicking with more memory to boot!


3:14 pm

Memory Upgrades for servers Pound, Gloop and Sunset

Posted (December 8th, 2006 at 3:14 pm PST) by mir

Both could use some more brainpower so we’re going to get them done today, the downtime should be minimal (under 20 minutes at the most).

– UPDATE Fri Dec 8 14:51:18 PST 2006 –

All three servers are back up and running with 4 GB of RAM apiece!


2:32 pm

Server Taffy Is Sticky Needs Replacement – Taffy back up

Posted (December 7th, 2006 at 2:32 pm PST) by Ralph

The server Taffy has been having network problems and has needed reboots. We have evaluated the machine while it is experiencing problems and we believe new hardware is required to fix the current issues the server is having. We are replacing the hardware right now and we will update this post as soon as the change is complete. Sorry about the problems for our taffy customers.

*****UPDATE*****

Sorry this took so long. After moving the server to new hardware the problem that was happening seemed to follow the machine. One of our administrators took a look at the server again and updated some of the networking configs. The server is up and working correctly now. Sorry this took so long to resolve, we’ll keep an eye on the server.


11:26 pm

Dedicated Server Moves

Posted (December 6th, 2006 at 11:26 pm PST) by Patrick

This is just a reminder to the dedicated customers that were notified last week of their servers being moved. Expected downtime is roughly 2 hours. Check this space for updates!

— UPDATE — Thu Dec 7 00:14:20 PST 2006

All servers are up in their new racks. We’re testing a few things still but it looks like everything survived the journey. Total downtime was approximately 1-1.5 hours. If you have any issues with your server please contact support.

Happy DreamHost Dedicated Distribution Team!


11:20 pm

Server moves (completed)

Posted (December 5th, 2006 at 11:20 pm PST) by DreamHost

We will be moving several of our webservers to make better use of our rackspace on Wed Dec 6, at 02:00:00 PST 2006. Downtime window is 3 hours, total downtime for each server within that window should be less than 30 minutes. This will not have any effect on your email, only your websites and FTP access will be affected. The servers being moved include:

apok
axl
balrog
blanka
brazil
charm
curly
dhalsim
dozer
egon
guile
limbo-looney
moe
neo
ryu
slappy
soy
tank
tofu
trinity
vega
venkman
winston
zangief
zuul

Update Wed Dec 6 04:34:05 PST 2006: All servers have been moved and are back online.


11:14 pm

File server shuffling (All Done!)

Posted (December 4th, 2006 at 11:14 pm PST) by DreamHost

Tonight we will be moving around 3 or 4 of our file servers. This is to make room for more space in our racks by optimizing how the servers are laid out. We’re sorry for the short notice. Downtime is about 1 hour if you are on an affected file server.

All Done! Total downtime wasn’t more than 5 minutes per file server. Only one of our regular web servers had to be rebooted. Everything should be back to normal, please let technical support know if you are having any problems!


12:12 pm

12.05.2006 – Mailman Upgrade

Posted (December 1st, 2006 at 12:12 pm PST) by beth

On Tuesday, December 5th, 2006, we will be upgrading Mailman at 3pm PST. This process will take approximately 3 hours, and most discussion lists will be inaccessible during this time. Discussion list E-mail will not be lost, and will resume shortly after. This will only affect discussion lists. No other services will be affected by the mailman upgrade.

More info:
This upgrade is to version 2.1.9, the most recent. It’s mostly bug fixes and maybe a few minor new features. There has been some development on Mailman 2.2 which should dramatically improve performance for systems like ours with a lot of hosted discussion lists. Mailman 3.0 is still a long ways out from what I have seen of the development process.


 
© 1996-2006, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
5:41 pm

ns2.dreamhost.com down for short while

Posted (November 30th, 2006 at 5:41 pm PST) by jeremy

Our offsite DNS server, NS2 is currently offline due to an apparent attack against the facility it is located in. We’re working with the provider to get more information regarding an ETA and a more solid cause for the outage. Until then, you may experience slow connections to some of your sites.

— UPDATE 2006-12-01 13:17 PST —

After speaking with the network engineer at our secondary facility it appears (as of this morning anywawy) that the attack is still under way. Their upstream providers say we’re still getting upwards of 30Mb/s in traffic towards our NS2 IP. They’ll be checking on it again shortly and hopefully it will have subsided enough to get us back online. We apologize for this inconvenience and appreciate your patience while we get this sorted out.

— UPDATE 2006-12-01 12:39 PST —

It looks like this problem is fully resolved now. Sorry for the late notification, but we wanted to be sure the problem was fixed and not just an intermittent patch.


1:12 am

Another Batch of RAM Upgrades (Completed)

Posted (November 30th, 2006 at 1:12 am PST) by Patrick

The following servers will have their BIOS flashed and memory upgraded:

union, twizzler, herod, patton, and io.

Union and twizzler are up first! Expected downtime is ~20 minutes.

– UPDATE Thu Nov 30 00:28:35 PST 2006 –

Both union and twizzler are back up and running. Herod and patton are going down now.

– UPDATE Thu Nov 30 00:44:16 PST 2006 –

Herod and patton are upgraded and operating properly. The last server to be upgraded tonight is io. Being that it’s across the hall in a different datacenter, it will not be going down for another 10 minutes.

– UPDATE Thu Nov 30 01:03:10 PST 2006 –

The final upgrade, io, has been completed.


10:29 pm

Renewed attack against Jalapa

Posted (November 29th, 2006 at 10:29 pm PST) by DreamHost

We’re working to fix it right now.

The same servers may experience slowdowns or brief outages until we have this resolved:
burton
euclid
hoover
highland
locust
pico
venice

Update; 10:02pm pdt We’ve stopped this attack and are in the process of investigating to find the source and target, in hopes of preventing it from happening again.


11:05 pm

Server Jalapa under attack!

Posted (November 23rd, 2006 at 11:05 pm PST) by micah

The server jalapa is being hit with an unknown Denial Of Service attack at the moment which is saturating its switch.

This switch saturation is in turn affecting the following servers:

burton
euclid
hoover
highland
locust
pico
venice

We’re working on finding the target IP for the attack so we can stop it. Once that’s done we’ll post an all clear.

Sorry about the inconvenience, it appears that not even Thanksgiving is safe from internet troublemakers :(

— UPDATE —
The offending IP has been null routed at our core router and things are returning to normal. We apologize again for the inconvenience.

Happy DreamHost Turkey Day Team!


12:50 pm

Server Charm is having networking problems. (UPDATED)

Posted (November 22nd, 2006 at 12:50 pm PST) by David

Our admins are working on correcting the situation as quickly as possible. We apologize for the inconvenience. Check back here for updates.

Update – The problem has been resolved. Everything should be back to normal again. If you have any problems with this server please open a support request via the control panel. Thanks for your patience.


12:44 am

More Memory Upgrades

Posted (November 22nd, 2006 at 12:44 am PST) by Patrick

The following servers will have their RAM upgraded over the next couple of hours:

caesar, wasabi, rexford, alameda, vermouth, bourbon, and leeloo

Not all of these machines require a BIOS flash like last night, but a few of them do. We’ll knock those out of the way early. First up, caesar and wasabi!

– UPDATE Tue Nov 21 23:56:31 PST 2006 –

Both caesar and wasabi are back up and running nicely. Next up, rexford and alameda…

– UPDATE Wed Nov 22 00:17:14 PST 2006 –

Rexford and alameda are both back up and running happily. Finally, vermouth, bourbon, and leeloo take the plunge.

– UPDATE Wed Nov 22 00:31:03 PST 2006 –

The last three RAM upgrades went smoothly. Vermouth, bourbon, and leeloo are all back up and running properly.


8:33 pm

Postac Hardware Issues

Posted (November 21st, 2006 at 8:33 pm PST) by Javier

The postac server is having problems right now and we are moving it to new hardware. We will post an update once we are done.

Postac has been completely moved to new hardware (around 8pm actually) if you’ve still got any problems with sites hosted there please contact support.


12:20 pm

File server swap (Done!)

Posted (November 21st, 2006 at 12:20 pm PST) by DreamHost

We are swapping out the “head” half of one of our file servers in about 15 minutes. The management interface is showing warning signs of an impending failure, and so we need to do this as soon as possible. This is to prevent an emergency swap later. Downtime should be about 10 minutes, likely less.

All done! Everything looks normal so far, but we will not know for a couple of hours if the transient error is gone.


12:32 am

Memory Upgrades

Posted (November 21st, 2006 at 12:32 am PST) by Patrick

The following servers are all having additional RAM installed tonight:

bundy, van, homer, hill, yoda, fairfax, and peso.

First up are bundy, van, and homer! These three servers are going down now to have their BIOS flashed and their memory upgraded. Total expected downtime is 20 minutes.

– UPDATE Mon Nov 20 23:57:44 PST 2006 –

Bundy, van, and homer are all back up and running. Both hill and yoda are going down now for their upgrades.

– UPDATE Tue Nov 21 00:12:54 PST 2006 –

Hill and yoda are both back up anr running properly. Fairfax and peso will be going down momentarily.

– UPDATE Tue Nov 21 00:36:03 PST 2006 –

All machines are back up and running properly with a greater allocation of RAM.

– Quick add-on update! Tue Nov 21 01:39:51 PST 2006 –

Slaw is getting thrown into the mix as well! It’s going down now for a quick upgrade.

– Final Update Tue Nov 21 01:48:02 PST 2006 –

Slaw is back up and running properly.


8:42 pm

Soy Hard Drive Error – Updated

Posted (November 18th, 2006 at 8:42 pm PST) by Terri

Soy’s hard drive just failed. We are in the process of moving everyone to new hardware. This should take no longer than an hour. We apologize for the inconvenience.

Updated: Soy is back online. All services are up and kicking.


 
© 1996-2006, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
1:14 am

Whopper getting DOS’d (Not anymore!)

Posted (November 15th, 2006 at 1:14 am PST) by DreamHost

It seems whopper is getting dos’d (possibly ddos’d). It is currently doing about 120k packets per second and totally saturating the uplink on the switch. I am working on isolating the source so I can drop it at our router and upstream if needed. More on this as it develops.

These servers are affected:

grizabella snowdrop lincoln fletcher gaz fairfax firestone paramount holt lifesaver whopper

All is better (mostly!) We have isolated the destination, however the source is pretty well distributed. Whopper got a whopper for a ddos. (I couldn’t resist…) For now there are a handful of domains which are offline while we investigate which one was the target, however the rest of whopper and the other servers named above should be OK now!


5:40 am

Quick file server move (all done!)

Posted (November 14th, 2006 at 5:40 am PST) by DreamHost

We need to move a file server from one rack to another. The total downtime should be less than 10 minutes, sorry about the short notice!

The move is complete. Any interupted services should resume shortly!


2:01 pm

More brains for Melrose!

Posted (November 10th, 2006 at 2:01 pm PST) by mir

After inspecting the machine for a while after having to restart it I noticed that it was rather low on memory – Patrick being the fine specimen of a human being that he is has run down to upgrade that for us right now. It should be back up within 10 minutes and healthier than ever (I did also find and firewall some naughty IPs so it should be much more stable now).

– UPDATE Fri Nov 10 13:08:22 PST 2006 –

Melrose is back up and running with more marbles, errr, memory!


11:33 am

Ganymede web server being moved to new hardware

Posted (November 10th, 2006 at 11:33 am PST) by Justin@DH

The ganymede web server is being moved to faster and newer hardware. Services are already starting to come back up, I’ll post an update when finished.

Update: Move complete and service restored.


3:25 am

Power sag in our primary datacenter (Update! Again!)

Posted (November 10th, 2006 at 3:25 am PST) by DreamHost

We just experienced a power sag in our primary datacenter. The affects of this are being investigated at this time. (It happened <4 minutes ago.) We know that many servers rebooted and are on their way back up on their own right now.

Sorry for the delayed update. After some finagling with our internal database servers, a switch failing, and other fun stuff, service is coming back up to a more normal level. We are going to continue to monitor the sitation and see how things are looking. We will post a final update soon!

We believe we are up past 95% of services restored. If you are still having problems please submit a technical support request through our web control panel:

https://panel.dreamhost.com/index.cgi?tree=support.msg

We apologize for the downtime, and the extended time bringing service back up.


4:04 am

Quick battery swap in a file server (Done!)

Posted (November 9th, 2006 at 4:04 am PST) by DreamHost

A backup battery just went in our file servers. I will be bringing it down shortly to swap it out. No data will be lost and expected downtime is 5 minutes.

All done!

Thu Nov 9 11:12:15 GMT [rc:notice]: The system was down for 174 seconds


3:22 am

The little Squirt

Posted (November 9th, 2006 at 3:22 am PST) by DreamHost

Squirt was down this morning at 2am for about 15-20 minutes. NFS was wreaking havoc, and the server was power cycled and has regained normalcy.

I’m sorry for the inconvenience to the users on squirt.


1:43 am

File server software problem (Update!)

Posted (November 7th, 2006 at 1:43 am PST) by DreamHost

One of our file servers was rebooted to clear an error with its floppy drive, of all things, and is not properly coming back online. We are working on identifying the problem and resolving. No ETA currently.

The file server is now booted up. I swapped out the bum floppy drive and it booted up on an older version of the OS. I do not know why, but that floppy drive was causing some weird errors in other parts of the system, so it could have prevented the updates from being written. I made and booted up off the 9 floppy disks the OS takes up, only to have disk 7 be bad, remade it on a new floppy and the machine booted up just fine.

I will now be researching how to ensure the OS on the hard drives is the correct one to prevent this in the future, and fixing any crashed servers this may have caused.

Tue Nov 7 09:46:17 GMT [rc:notice]: The system was down for 4145 seconds


12:09 am

One-click Updates – Mediawiki, Gallery, and Zencart

Posted (November 7th, 2006 at 12:09 am PST) by DreamHost

I’ve updated the the One-Click Installs for MediaWiki, Gallery, and Zencart, after repeated threats from Josh that he would bludgeon me with a Guitar Hero controller unless I did so. Here’s the changelogs:

MediaWiki 1.8.2
Gallery 2.1.2
Zencart 1.3.6

If you installed any of these applications using our One-Click Installer, you can upgrade it by logging in at panel.dreamhost.com, clicking on the “Goodies” tab and then on the “One-Click Installs” page, and then clicking on the upgrade link next to the appropriate application installation.


5:25 pm

Possible brief network outage tonight at 6PM

Posted (November 6th, 2006 at 5:25 pm PST) by jeremy

In our continuing effort to expand safely, we need to bring a new circuit online. There’s a possibility of network interuption for a small part of our network, but we will have our Cisco consultant on hand just in case. Only a small portion of our network may possibly be affected for a short period and affected services will include POP3/IMAP, SMTP, web, and jabber.

Happy DreamHost Grabbing-the-Live-Circuit Team!


 
© 1996-2006, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
6:46 am

Ike is dying slowly

Posted (November 6th, 2006 at 6:46 am PST) by DreamHost

Well boys and girls, I’ve got some bad new. Ike is slowly dying, it seems to be under a DDoS (http://en.wikipedia.org/wiki/Ddos) attack, bad enough that no one can get a shell. An Admin is on his way out to the scene of the crime, and should be there in the next 20 minutes with his magnifying glass and pipe. I estimate a 50-90 minute downtime at this point, and I’m sorry for the inconvenience it’s causing.


8:52 am

alameda crashed (Back up!)

Posted (November 4th, 2006 at 8:52 am PST) by DreamHost

Alameda crashed a little while ago. Attempts to revive it are not going well, and it seems the power strip is not responding to remote commands. A technician is on their way in right now to reboot the server. ETA is 30 minutes just to be safe, it will hopefully be faster than that.

I’m sorry about the delayed update. Alameda has been back online for about 1 hour now. Please email technical support if you are still having problems!


2:49 am

More RAM Upgrades (Complete!)

Posted (November 2nd, 2006 at 2:49 am PST) by Patrick

The following machines are having their RAM upgraded:

curly, salt, soda, ouzo, cordial, gimli, cornelius, crenshaw

They will be upgraded in three batches. First curly, salt and soda. Second ouzo, cordial, and gimli. Finally, cornelius, crenshaw, and zorg.

The first batch (curly, salt and soda) is going down for the upgrade now. Check this space for updates.

– UPDATE Thu Nov 2 02:05:36 PST 2006 –

curly, salt and soda are all back up and screaming along! ouzo, cordial and gimli are up next!

– UPDATE Thu Nov 2 02:30:24 PST 2006 –

ouzo, cordial, and gimli are all back up and running properly. cornelius and crenshaw are up next!

– UPDATE Thu Nov 2 03:00:08 PST 2006 –

cornelius and crenshaw are both back up and running. If you have any problems whatsoever with the aforementioned servers, please contact support.


3:31 pm

Quick battery swap in a file server (Done!)

Posted (October 31st, 2006 at 3:31 pm PST) by DreamHost

Another battery just went in our file servers. (Different server!) I will be bringing it down shortly to swap it out. No data will be lost and expected downtime is 5 minutes.

All done!

Tue Oct 31 22:38:33 GMT [rc:notice]: The system was down for 193 seconds


2:19 pm

Analog Web Stats Reporting Changes

Posted (October 31st, 2006 at 2:19 pm PST) by Dallas

To reduce overall server resource consumption we will be making a change to how we run our Analog-based website statistics reports. Currently, we generate three reports on a daily basis. The reports are called ‘Daily’ (the previous 24 hours of data), ‘Monthly’ (the current month to date), and ‘Longterm’ (limited reporting of all of the historical data we have). The data in the Daily and Monthly reports is largely redundant so we will stop generating the Daily report altogether. The other two reports (Monthly and Longterm) will still be generated daily. This change will start going into effect next week.

If you would like us to continue generating the Daily web stats report for your domain(s), you can enable that setting from this section of our web control panel…

https://panel.dreamhost.com/index.cgi?tree=status.stats&

Click the ‘Customize’ link next to the domain to see the options available.

Contact our support team if you have any questions about this.


10:21 pm

Server Dollar NOT money….moving to new hardware

Posted (October 30th, 2006 at 10:21 pm PST) by glen

Hardware failure with server Dollar is keeping it from loading it’s service. It is being moved to new hardware right now. Should be completed in under an hour. Update to Follow

UPDATE
All Apache’s are back up. If your site is still down on the server Dollar please submit a support request through the panel.


3:05 pm

Quick battery swap in a file server (Done!)

Posted (October 30th, 2006 at 3:05 pm PST) by DreamHost

We need to do a quick battery swap in the file server that failed this morning. This is typically a 5 minute procedure.

All is back to normal with this file server:
Mon Oct 30 22:14:50 GMT [rc:notice]: The system was down for 203 seconds

Sadly not my best time (188seconds), but still pretty good. Those darn screws wouldn’t line up correctly.


9:05 am

File storage server issues

Posted (October 30th, 2006 at 9:05 am PST) by andrea

One of our many file storage servers suffered an outage which has caused high loads and slowdowns on some servers (only affected one cluster/group of servers). The storage machine is back online, no data was lost. You may experience reboots, high loads and slowdowns on your machine for a short while longer, but everything is on its way back to normal.

Very sorry about the inconvenience. Please contact support if you have any questions.


1:13 am

WordPress One-Click Install updated to version 2.0.5

Posted (October 30th, 2006 at 1:13 am PST) by DreamHost

I’ve just updated our One-Click Install of WordPress to version 2.0.5. The bevy of security fixes and new features can be read about here:

Changes In WordPress 2.0.5

If you installed WordPress using our One-Click Installer, you can upgrade it by logging in at panel.dreamhost.com, clicking on the “Goodies” tab and then on the “One-Click Installs” page, and then clicking on the [Upgrade to 2.0.5] link next to your installation of WordPress.


9:57 pm

Selma denial of service attack

Posted (October 27th, 2006 at 9:57 pm PST) by DreamHost

I apologize for not posting this earlier. It looks like selma was under a large denial of service attack. It was receiving about 100mbit of traffic from a large number of hosts. The major offendors have been blocked at our firewall and we are working on identification of the target, as well as restoring service for the handful of websites which are still offline.


 
© 1996-2006, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
5:17 pm

Rhod being moved to new hardware (Resolved!)

Posted (October 26th, 2006 at 5:17 pm PST) by DreamHost

Rhod ran into some troubles and had to be moved onto bigger, better hardware. We’re working on this issue as quickly as we can, and will post an update as soon as we’re finished.

– UPDATE Thu Oct 26 07:00:56 PDT 2006 –

Rhod is back up with new hardware!


3:31 am

RAM Upgrades (Complete)

Posted (October 26th, 2006 at 3:31 am PST) by Patrick

More memory upgrades! This time it is:

malt, harpo, larchmont, and absinthe

Followed by:

ord, garth, and bixel!

Check this space for updates.

– UPDATE Thu Oct 26 04:01:14 PDT 2006 –

malt, harpo, larchmont, and absinthe are complete!

ord, garth, and bixel coming up!

– UPDATE Thu Oct 26 04:40:57 PDT 2006 –

garth, and bixel are all updated and back up. Unfortunately ord’s hardware didn’t take to the BIOS flash necessary for the new memory. It is being moved over to new hardware now and will be back up soon.

– FINAL UPDATE Thu Oct 26 06:06:31 PDT 2006 –

ord is back up and running on new hardware. If you are experiencing any problems whatsoever, please contact support.


5:40 pm

Webmail attachments problem

Posted (October 25th, 2006 at 5:40 pm PST) by DreamHost

The folder where we temporarily store webmail attachments filled up. It seems that Squirrelmail was leaving stale files around, and it hit the maximum directory size. We are cleaning these out now. We will look in to why this was happening to try and prevent it in the future.


2:09 am

Tank Puckered Out (Resolved)

Posted (October 25th, 2006 at 2:09 am PST) by Patrick

It put up a good fight, but in the long run, tank lost the war. It’s being moved over to new hardware now to fight another day. Check this space for updates.

– UPDATE Wed Oct 25 02:59:56 PDT 2006 –

Tank is back up on new hardware and is ready for battle! If you are still having problems with tank, please let support know.


2:51 pm

Go Hardware Failure (Resolved)

Posted (October 24th, 2006 at 2:51 pm PST) by Terri

A network interface card has failed on machine go, and we are working right now to switch everyone over to new hardware. We apologize for the inconvenience. We will have everyone back up and running within an hour.

Edit: Oct. 24 3:42 PM PST. Go and all its services are back up and running.


2:17 am

More Server Relocating (Complete)

Posted (October 24th, 2006 at 2:17 am PST) by Patrick

The following servers are going to joing their comrades in Alchemy (the datacenter across the hall):

kiki, limbo-postal, gap, medina, lira, yerba, dali, spree

Again, expected downtime is around 20 minutes. As always, check this space for updates.

-UPDATE Tue Oct 24 03:04:06 PDT 2006-

The servers have been moved and are all back up. If you are having any problems with one of these servers after the move, please contact support.


11:54 pm

Server Relocation (Complete)

Posted (October 23rd, 2006 at 11:54 pm PST) by Patrick

The following six servers are being relocated:

ruble, io, zuul, slash, squirm, rhod

They are moving across the hall to our other datacenter. Total downtime will be around 20 minutes. Check this space for updates.

-UPDATE Tue Oct 24 00:49:30 PDT 2006-

Everything is back up and running. If you are having any problems with your machine, please let support know.


3:14 pm

Wanna Fanta!

Posted (October 21st, 2006 at 3:14 pm PST) by Brian@dreamhost

It looks like Fanta’s network interface fizzled out, so, we decided to restore it to new hardware.

All of the network configuration scripts are furiously working to restore it’s … pop … so we should have it back up and working soon! Sorry about that!

Happy-not-soo-bubbly-server-restore-team!


4:40 pm

Fileserver switch troubles (Resolved!)

Posted (October 19th, 2006 at 4:40 pm PST) by DreamHost

One of the switches which connects some of our fileservers is having unexplained high cpu load. This is causing slower than normal loading of some sites. We’re looking at the issues and hope to have it fixed up shortly. Updates to follow.

We think we have this one resolved. It seems the CPU load was simply too high and causing the switch to have watchdog timeouts and reboot. This was affecting about 8-9 file servers. We installed another switch to handle the extra load and watching it for half an hour saw no problems. We will continue to monitor it throughout the night to ensure that the problem is completely resolved.


9:49 pm

RAM Upgrades! (Completed)

Posted (October 18th, 2006 at 9:49 pm PST) by Patrick

Mayo, wilshire, and cognac are all going down for RAM upgrades now. Unfortunately they need to have their BIOS flashed as well, so downtime will be a tad bit longer than normal (around 20 minutes). Check this space for updates.

– UPDATE Wed Oct 18 22:10:59 PDT 2006 –

All three servers are flashed and back up with double the RAM!


 
© 1996-2006, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
3:11 pm

File server failure (Resolved)

Posted (October 17th, 2006 at 3:11 pm PST) by micah

One of our file servers failed about 30 minutes ago causing high loads and site inaccessibility for some of you. We are currently swapping in new hardware so everything should be online shortly. The disk shelf is fine so there won’t be any lost data.

We’ll post an update as soon as everything is back online.

We have swapped out the hardware and the file server appears to be running again. We are working now to restore service on affected servers. No data was lost in this failure.


1:26 pm

Nelson is Being Moved to New Hardware

Posted (October 17th, 2006 at 1:26 pm PST) by Justin@DH

One of the network cards in the nelson server has failed. The server is being moved to new hardware now and will be back up shortly. Our apologies for the downtime.


9:24 am

File Server Moving!

Posted (October 17th, 2006 at 9:24 am PST) by DreamHost

More File server moving. This is for Wednesday night, October 18, 2006 starting at 10pm sharp! Again, we have no easy way of saying who will be affected, however less than 10% of our total customer base.

I have an idea for how to make this go twice as fast, so hopefully it will be done a full hour sooner than usual. Downtime window is 10pm to 1am per usual.


8:24 pm

Panel down (and back up)

Posted (October 16th, 2006 at 8:24 pm PST) by jeremy

Our private network has had some unexpected issues that we’re currently working on. Until they are resolved, the panel and related updates will be down. We’ll post more information as we receive it.

As of 8:45pm PDT, the panel is back up. It looks like our firewall had some issues requiring a reboot. We received our replacements today and we’ll be configuring them this week, so this should be behind us very shortly.


6:07 am

Brazil is Being Moved to New Hardware (Complete)

Posted (October 16th, 2006 at 6:07 am PST) by Patrick

The hardware that brazil is on failed. It is being moved over to a new server now and will be back up soon. Check this space for updates.Mon

-UPDATE Oct 16 06:31:07 PDT 2006 –

Brazil is up and running on new hardware and websites are coming up now.


6:41 pm

Panel Issues and One-Click Install Emails

Posted (October 15th, 2006 at 6:41 pm PST) by glen

The issues with delayed user creations and one-click Installs has been cleared up by our admin team. During this fixing and troubleshooting process there was a 2nd bug created that effected many users. One-Click install emails were dispatched to a large amount of customers, despite the fact that no changes had been made on their account . If you haven’t used the one-click system in the last few days, you can safely just disregard it.


9:54 pm

Webpanel Issues

Posted (October 13th, 2006 at 9:54 pm PST) by DreamHost

There are some problems creating emails and doing one-click installs. We’re currently investigating this, and hope to have it resolved shortly, we’ll post an update when we do.


3:27 pm

Mail database maintenence (completed)

Posted (October 10th, 2006 at 3:27 pm PST) by DreamHost

Several of our databases which are used by our mail system will be going offline for maintenence early tomorrow morning, Wed Oct 11 02:00:00 PDT 2006. Expected downtime is 30 to 60 minutes. This will take DreamHost webmail for all domains offline during that time. Webmail should be the only service interrupted by this, and mail will continue to be delivered while it is offline. IMAP and POP3 access will remain online the entire time.

Update: Wed Oct 11 02:48:30 PDT 2006: All maintenence completed. Webmail is back online.


2:52 pm

File server shuffling

Posted (October 10th, 2006 at 2:52 pm PST) by DreamHost

Tomorrow night, 10/11/2006, at 10PM Pacific we will be moving 4 or 8 file servers into new positions in their racks. This is the same as previous moves, and is to increase their density/optimize their placement in the racks. This will affect most clusters. Downtime is estimated to be about an hour, however you will likely only experience it if you have files actually on these servers.

No email will be lost during the outage.


2:29 pm

Simplified Announcement Categories

Posted (October 10th, 2006 at 2:29 pm PST) by Dallas

We’ve changed our categories for announcements here a little bit to hopefully clarify some of the information being announced. The new categories are mostly severity based and reflect how large of an impact on our services a particular announcement has. This is still a work in progress so feel free to comment on how effective or not you think this change will be.


 
© 1996-2006, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
12:39 pm

Tequila being moved to new hardware (fixed!)

Posted (October 10th, 2006 at 12:39 pm PST) by DreamHost

We are in the process of moving tequila over to new hardware. It seems something is wrong with the networking. This affects your ssh, ftp, web and jabber services. Email and mysql is unaffected. Things should be up and running in under an hour.

All fixed! It seems the network problems were actually related to a specific site. We have isolated this site and fixed the problems.


1:32 pm

RAM Upgrades!

Posted (October 9th, 2006 at 1:32 pm PST) by Patrick

The following servers are going down for memory upgrades:

olympic
zuma
absinthe
pretzel
cognac
mayo
bellflower
wilshire
otto

They will go down three at a time (in order) for the upgrades, which will take roughly 10 minutes per 3 server set. Updates will be posted as each three server set is back up. First up…

olympic, zuma, and absinthe!

– UPDATE Mon Oct 9 13:59:05 PDT 2006 –

olympic, zuma, and absinthe are back up and running. Absinthe’s hardware did not agree with the memory upgrade however and is still in need of an upgrade. This will be dealt with at a later date.

Pretzel, cognac, and mayo are going down now.

– UPDATE Mon Oct 9 14:27:53 PDT 2006 –

Pretzel is upgraded. It appears that one model of server (although being almost identical hardware from the same vendor) does not agree with the RAM upgrades. This is the case with absinthe, cognac and mayo. Wilshire is also the same hardware and will not be upgraded at this time.

Bellflower and otto going down for upgrades now.

– FINAL UPDATE Mon Oct 9 14:40:17 PDT 2006 –

Bellflower and otto are back up and running properly!


8:23 am

Database server odie, hardware troubles.

Posted (October 7th, 2006 at 8:23 am PST) by DreamHost

The database server ‘odie’, is having hardware issues which have impacted its stability. Those of you who have recently signed up, and are having recurring database troubles, this is more than likely the cause. We’re in the process of migrating everyone’s data to a different server, and will update this post once it has completed.

Update Sat Oct 7 10:06:31 PDT 2006: All databases have been migrated from ‘odie’ onto a new server called ‘ren’.


2:56 pm

Problem with Aviation Server

Posted (October 6th, 2006 at 2:56 pm PST) by Dallas

We are aware of the problem with the ‘aviation’ web server and are working on resolving it. The issue appears to be specific to just that server. The main symptom is very slow loading websites. This is not related to any mysql, email, or any other issue other than aviation itself.

Update: 3:28PM PST

Aviation is currently being moved to new hardware. We belive the problem to be due to network cards going bad. Expected time for service to return to normal is about 15-20 minutes.

Update: 5:22PM PST

Aviation was back up as of about 4:00PM PST and has been working normally since then. No other servers were affected by this issue and any other problems you have be experiencing are unrelated. If you are experiencing a service issue, please make sure you have contacted our support team.


1:46 pm

Switch Update (Selma server issues)

Posted (October 6th, 2006 at 1:46 pm PST) by micah

The issues that arose yesterday haven’t surfaced again. The affected servers were venkman, cerritos, archibald, flanders, rupee, ralphie, cheeky, pizarro, butterfinger, and selma and they all seem to be doing much better with the exception of selma.

Selma was moved to an entirely different rack last night to see if we could isolate the problem. It crashed a couple more times and this morning we discovered an IP that was crippling selma with an attack of some sort. We’ve since firewalled that IP and selma seems to be doing much better, we’re keeping a close eye on it (as well as the previously affected switch) to make sure these troubles are behind us.

We appreciate your continued patience through these strange issues, if I haven’t specifically mentioned your server above then any problems you are having are unrelated and you should contact our tech support team.


6:37 pm

Switch Update

Posted (October 5th, 2006 at 6:37 pm PST) by micah

We’ve done a couple things since the last post regarding the malfunctioning switch. The switch firmware has been upgraded and the final reboot this entailed seems to have brought everything back up in normal working order. We’ve isolated a few suspect machines to see if the problems follow them or if it is indeed just a network hardware issue.

Our techs and admins are standing by tonight in case this issue pops up again.


5:12 pm

More Switch Troubles

Posted (October 5th, 2006 at 5:12 pm PST) by micah

And they seem to be following some of the servers…

A switch in our Alchemy datacenter is not communicating with the rest of our network. This is affecting the following servers:

venkman, cerritos, archibald, flanders, rupee, ralphie, cheeky, pizarro, butterfinger, and selma

These servers will have intermittent connectivity to the rest of the internet and to their file servers which will result in website slowdowns and outages for some of you.

We are investigating the cause now and will have an update for your shortly. This is extremely similar to a few recent incidents (though on completely different switches) and we’re investigating thoroughly. We apologize again for the inconvenience this causes. Everything should be back to normal shortly.


1:12 pm

Packet Loss

Posted (October 5th, 2006 at 1:12 pm PST) by Ralph

We are currently experiencing packet loss over one of our incoming network links. Our administration team is currently working on this issue. We will update this status post once we have more information. Sorry about any problems you are experiencing.

UPDATE: This issue has been resolved now. This was occuring for approximately 15 minutes. The problem would have manifested itself as network slowness. If you are experiencing any other issues please contact support.


2:54 pm

Web Server “selma” Died (UPDATE)

Posted (October 4th, 2006 at 2:54 pm PST) by micah

The server “selma” crashed pretty hard over an hour ago and all attempts at reviving it have failed. We’re currently swapping in new hardware for it and everything should be up and running again shortly.

UPDATE 10/6/2006 6:00am PDT (GMT-8:00) Selma server is having problems with some of the Apache processes not running properly. If you have domains hosted on this server some may work and others may not. We are working to correct this problem as quickly as possible. Please check back later for further updates on the situation. I apologize for the inconvenience.


5:40 am

Switch Troubles

Posted (October 4th, 2006 at 5:40 am PST) by Patrick

A switch in our Alchemy datacenter is not communicating with the rest of our network. This is affecting the following servers:

alexander, baht, contra, smithers, lenny, mcclure, millhouse, willie, seamus, selma, abbazaba, defender, joust, altoid, lisa, skinner, chalmers, wiggum

We are investigating the cause now and will have an update for your shortly.

– UPDATE –

The switch had dropped many of the MAC addresses for the servers connected to it, leaving packets with nowhere to go. This has been resolved and a majority of the servers are back up. There are a few stragglers that are still being worked on. Thank you for your patience and everything will be up 100% again soon.


 
© 1996-2006, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
11:05 pm

RAM Upgrades (Complete)

Posted (October 3rd, 2006 at 11:05 pm PST) by Patrick

Pico and attila are going down for RAM upgrades. Total downtime will be roughly 10 minutes. Check this space for updates.

– UPDATE Tue Oct 3 23:15:12 PDT 2006 –

The upgrades are complete and both servers are back up with double the RAM.


3:50 pm

First chance to donate to Defective By Design and Santa’s Helpers.

Posted (October 3rd, 2006 at 3:50 pm PST) by Josh

Well, you missed it. We’ve switched charities now on the web panel to Defective by Design and Santa’s Helpers.

(We ended up getting $904.21 from 36 contributors for the WWF and $383.32 from 17 contributors for C3.) Whoa.


1:07 pm

Webmail/E-Mail connection problems

Posted (October 3rd, 2006 at 1:07 pm PST) by Ralph

Some of you are experiencing intermittent problems when connecting to your email. The majority of you are noticing it with webmail, but it is an intermittent problem affecting pop and imap connections as well. We believe these problems are only occuring on one cluster (spunky) which we are investigating. We are working on these problems now. If you are experiencing intermittent connection issues please send in a support message to dhstatus@dreamhost.com.

Update: We tracked the problem down to a mail database clone which was not properly updating, we have removed this until it has been fixed and the errors have gone away. We are also working to optimize the mail server connections to the mail database machines which should eliminate any remaining issues (specifically the intermittant SMTP errors when sending).


9:30 am

Last chance to donate to the WWF and CCC!

Posted (October 2nd, 2006 at 9:30 am PST) by Josh

Hey,

This is just a note to let everybody know we’ll be switching our charities tomorrow.. so today is your last chance to donate to the WWF and/or the Colorectal Cancer Coalition via our Charity Page and get your donations matched by DreamHost!

So far we have 14 donations to CCC for a grand total of $333.32 and 33 donations to the WWF for $814.21. Let’s pump those number UP..


9:29 am

Problems with networking (No really this time)

Posted (October 1st, 2006 at 9:29 am PST) by jordan

We are currently having some problems this morning with a switch and are trying to move machines off of that switch right now. Downtime includes customers on servers cerriots, hernod , rossmore , oxnard , arrow. We apologize for the trouble and hope to have the switch replaced asap.

—-
We have moved the servers in the affected rack to knew switches in other racks temporarily while we get a new switch in place. We are working on recovery efforts now on servers which may be down. The list of affected servers is:

Arrow, mel, caesar, herod, alondra, overland, rossmore, oxnards, cerritos, selma, nala, demeter, jarvis. This is a mix of 3 MySQL servers and the rest webservers.

Sorry about no timestamp before, it’s 12:36 Pacific time (-0800? I never could keep track of daylight savings time.)

We have completely replaced the switch and are working on getting the servers back online. All networking cables are back in their regularly scheduled switch ports.

Wierdness is afoot. The servers are still exhibiting the same problems, we have even moved two of them into our new datacenter and it’s the same deal. It’s only specific servers in half of one of our racks.

As of 1700 Pacific time (GMT-0800 or so, 5PM), we think the issues with this rack are behind us. We moved some machines to new hardware and this seems to have fixed the problems. We will be looking in to this more on monday, and doing some extensive testing on that hardware to make sure it was the root cause of the problem.


9:27 am

Quick battery swap in a file server (Done!)

Posted (September 30th, 2006 at 9:27 am PST) by DreamHost

As long as I’m down here at 9:30am on a saturday… :)

I need to do a quick NVRAM battery swap in one of our file servers. This should only take about five minutes. (This battery is what keeps all of your data 100% safe in the event of a power outage or other hardware failure!)

Sat Sep 30 16:32:33 GMT [rc:notice]: The system was down for 188 seconds

Oh yeah. Your data is (even more) safe, once again!


8:30 am

MySQL machine ‘demeter’ down (Not anymore!)

Posted (September 30th, 2006 at 8:30 am PST) by Karl

Demeter didn’t want to come back after a reboot. An admin is en route to the datacenter now to remedy the problem. No ETA at this time, but we’re hoping no more than 1-2 hours.

Looks like it was just powered off. Its networking was being flaky, so I attempted a reboot, still nothing. Often times a powercycle will re-start the network cards properly, as they never go off due to Wake-on-LAN. Ran `halt` to bring the system down safely, and the thing powered off! None of our other servers do that. :(

The long powercycle seems to have done it, the databases are back online and showing their usual queries per second.

That will teach me to safely shutdown my machines before rebooting!


2:39 am

Database Server Moves: Rhubarb and Putney (Complete!)

Posted (September 28th, 2006 at 2:39 am PST) by Patrick

Both rhubarb and putney are being relocated. Both of these servers will be down for about 10 minutes during the rack re-shuffling. If your are unable to connect to your database, it is most likely hosted on one of these two servers. Check this space for updates.

-UPDATE (Thu Sep 28 02:53:16 PDT 2006)-

Both servers are relocated and back up and running. If you are still having a problem connecting to your databases, please contact support.


11:28 am

Moving three servers: barranca, van, runts

Posted (September 27th, 2006 at 11:28 am PST) by DreamHost

We will be moving the servers barranca.dreamhost.com, van.dreamhost.com, runts.dreamhost.com to a new rack at 12:00pm PDT (-0800). They will each be down for about 10 minutes. For those of you hosted on these servers, your web sites as well as FTP/shell access will be down during this time. Mail service will not be interrupted. Sorry for the short notice and inconvenience.

Update Wed Sep 27 13:23:57 PDT 2006: All three servers have been moved and are back online.


4:46 pm

Ryu’s kungfu is weak

Posted (September 23rd, 2006 at 4:46 pm PST) by DreamHost

Webserver Ryu has experienced some sort of hardware issue and is currently being moved onto new hardware. If you’re site is hosted on Ryu it should be up within an hour, we’ll post an update here when we have more news.

Fake update: Save != Publish. Ryu’s all up now, if any of your sites hosted on Ryu aren’t working please write to support and let us know.


 
© 1996-2006, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
1:06 pm

Network Problems Resolved

Posted (September 22nd, 2006 at 1:06 pm PST) by Ralph

We were making adjustments to our network that didn’t go as planned unfortunately so a number of you have noticed downtime or site/mail slowness again. That issue has been resolved. Sorry for the late announcement, the change was supposed to have no affect on our services. Everything is back to normal again. This problem started a little over an hour before this post and was set back within 20 minutes once we noticed we made a mistake. Sorry about the scare. If you feel you are still experiencing problems from this outage please write to dhstatus@dreamhost.com.


8:01 am

File Server Issue Causing a Few Web Service Interruptions

Posted (September 22nd, 2006 at 8:01 am PST) by Justin@DH

We have a file server that is filling up at an extraordinary rate and we are working currently to find the cause. During the search some web services may be interrupted. We are working to get this issue resolved as soon as possible.

– UPDATE –

The issue has been resolved and any servers that might have had web services temporarily turned off are being put back into action.

– Post Mortem –

This is what actually happened, to answer the various speculations below. Apache2.0 fixed the “bug” which meant that log files could not exceed 2.0G in size. We recently have been doing a lot of roll outs on Apache2.0, versus our previous default of Apache1.3. This meant, if a log file was growing at an enormous rate, it would cap out at 2.0gigs, crash the apache, and we could nearly instantly find and fix it.

Now, if a file is filling up at a rate of say, a gigabyte/minute, it can happily fill up a file server. We first noticed when a file server went from no warning on our monitoring software, to 12gigs free. We lowered the reserve space for our backups to give us breathing room, but still only had about 60 minutes to figure out what was causing this problem, and what to do about it. Running a quota report on the file server showed that root was using about 463gigs of space, well more than normal, so we assumed it was a log file growth issue. (All of our apache logs are stored by “root” for a few days so that we have accurate accounts of what happened on our servers.)

What this meant, was on one of the handful of servers that mounts the file server, in one of the thousands of home directories, there was a HUGE log file for us to delete. After some fruitless ideas of attepting to turn off Apache2.0 on servers which appeared to be “busier” than normal turned up nothing, we started running per-server “du” commands on all users log directories. (“du” stands for “disk usage”, it tally’s up how much space is being used by a file or directory regardless of user.) This went faster than expected, and on a hunch I skipped ahead to the server “flanders”, which turned out to be the culprit. Found the site, deleted the file, and went on our happy way restoring service.

[HomerSimpson]Stupid Flanders…[/HomerSimpson]

I apologise for the stopping some services to try and find the culprit. The idea behind it is if we can find the sevice which is stopping the growth, or even the server, we can prevent the file server from filling up, which would have put thousands more out of service. We are in the process of writing something to troll our file servers looking for these problems, however they tend to crop up very quickly.

Looks like we need to see if:

http://httpd.apache.org/docs/2.0/programs/rotatelogs.html

Is a viable solution for our logging needs.


4:23 pm

Large Mail Queue for Filtered Mail

Posted (September 21st, 2006 at 4:23 pm PST) by mir

If you are using our filtering system you may notice some delays in delivery, we have checked to make sure we were not spammed and were unable to find any specific senders abusing the system so mail is being allowed through (none has been blocked or deleted so it should not be lost) but may be delayed by several hours. We apologize for the inconvenience that this has caused.


1:44 pm

UPDATE on Snagglepuss

Posted (September 21st, 2006 at 1:44 pm PST) by brians

We had to replace the RAID card in the machine. The machine will be back up and running shortly. Again, we apologize for the inconvenience that the downtime has caused and appreciate your continued patience.


1:41 pm

Filer moving (Done!)

Posted (September 21st, 2006 at 1:41 pm PST) by DreamHost

One day I will remember to post this in a timely manner.

More filer moving! We have to continue our efforts to optimize our rack space to continue to feed your ravenous apatites for disk space. The downtime window starts at about 10pm tonight and goes until 1am at the latest. This will affect all clusters, however it should only affect your websites for a about 30 minutes if you happen to be on the physical filer we are moving.

Sorry about the short notice.


All done! Everything went really smoothly, none of the filers were down for more than 10 minutes. We have actually been done for about an hour now.


11:30 am

UPDATE on Snagglepuss

Posted (September 21st, 2006 at 11:30 am PST) by brians

Looks like we have to move snagglepuss over to new hardware. Worst-case scenario the machine will be down for 60-90 minutes. Since we ” might ” have to replace the drives as well. However, BEST-case is the machine will be up within the next 20 minutes or so. Of course we are striving for the latter. Again, we sincerely apologize for any inconvenience this may have caused.


11:16 am

Snagglepuss is down

Posted (September 21st, 2006 at 11:16 am PST) by brians

We are having problems with the machine that your database is hosted on. It was not responding to remote reboots. The Admins are in the datacenter checking on the machine as I type this post. Everything will be back up and running shortly. We apologize for any inconvenience this may have caused. You can check back here http://www.dreamhoststatus.com for future updates.


3:00 pm

Pico and Scotch Down for Memory Upgrades (Complete!)

Posted (September 19th, 2006 at 3:00 pm PST) by Patrick

Pico and scotch are going down for memory upgrades. They will be down less than 10 minutes. Check this space for an update on when they are back up and running properly.

-UPDATE-

Everything is back up and running!


1:24 am

Quick battery swap (All done)

Posted (September 19th, 2006 at 1:24 am PST) by DreamHost

I have to do an NVRAM battery swap in a filer really quickly. Downtime shouldn’t be longer than 5 minutes.

Update: All went well.

Tue Sep 19 08:49:56 GMT [kern.syslog.msg:notice]: The system was down for 264 seconds

Oh yeah.


4:52 pm

Network Issue Followup

Posted (September 18th, 2006 at 4:52 pm PST) by Dallas

It’s been a few days since our last change to our network and things are still looking very good. There are some isolated (and seemingly unrelated) issues we are dealing with, but the major network issue does appear to be resolved at this time.

This latest change has left our network setup in a not fully redundant state and we are working as fast as possible to fix that.

Additionally we are working to improve our power redundancy for our core network equipment, reorganize our network to provide more long-term scalability and improve performance, and further optimize the configurations on the new equipment we have already installed. Once everything is done we will have invested somewhere in the neighborhood of $300,000 on upgrades and improvements to our network, not to mention all of the human time involved. This is our highest priority.

Note that if you are currently having problems with your website it is not due to any known systemic issue. You are advised to contact our support team with details so we can look into it for you.


 
© 1996-2006, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
6:30 am

Server ‘pizarro’ has gone splat.

Posted (September 18th, 2006 at 6:30 am PST) by Karl

Moving over to new hardware now. Will update when this is complete.

Update 8:51 AM PST Pizarro’s been up for a fair bit now, but we’ve been busy cleaning up a few stray issues. The majority of sites on pizarro should be up now, and we’re working on the handful of stragglers.


3:03 pm

Network / Stability update!

Posted (September 17th, 2006 at 3:03 pm PST) by Brian@dreamhost

So far, things are looking much better as far as our network work goes. For now, we’re concentrating on any servers that may still be experiencing slowness, regardless of network performance.

If you are experiencing problems, please write us a support message with the subject “Server Slow Status” and we’ll get right on it.

This will help us narrow down any machines that need to be offloaded or otherwise troubleshot! =)


9:51 am

Rampart Scandal (Working!)

Posted (September 17th, 2006 at 9:51 am PST) by Patrick

Rampart decided to be scandalous and fail one of its ethernet ports. It’s being failed over to new hardware now and will be back up soon. Check this space for updates.

-UPDATE ( 11:19 PST )-

Rampart is up and running on new hardware. If you are having any problems with this transition, please let support know.


9:34 pm

Problems with Cheeky!

Posted (September 15th, 2006 at 9:34 pm PST) by Brian@dreamhost

We’re having a bit of a problem with some of cheeky’s web servers, so some domains on it may appear down. We’re currently working on the problem and will update as soon as it’s fixed!

Update Fri Sep 15 22:52:57 PDT 2006: Cheeky’s problems couldn’t be resolved on the old hardware, so we moved it to new hardware. It is fully online now, and the configs to get all the webservers are running currently, they should be finished soon.


3:18 am

Starburst hopefully more stable!

Posted (September 15th, 2006 at 3:18 am PST) by DreamHost

Just a quick note that I think I found what was causing starburst to crash over and over. Someone was illegally distributing copyrighted files in a very sneaky fashion, and causing apache to eat a -lot- of ram. At first (second, third…) glance it looked like the usual large amounts of -shared- ram that our stuff uses constantly. Two nights of banging my head against the server later I realized it wasn’t shared ram, when stopping that apache instance freed up 3.5 gigs of ram!

Either way, I think this server is ready to rock and roll. I will continue to monitor it for the next few hours, but the memory usage on it has bottomed out so I think we’re clear.


8:35 pm

Archibald Back UP – Failover Complete

Posted (September 14th, 2006 at 8:35 pm PST) by glen

Failover on Archibald to new hardware was a success. Sites are back up and stable. If your site is hosted on archibald and you see any errors / problems please submit a support ticket with as much detailed info as possible.


7:28 pm

Archibald Failover

Posted (September 14th, 2006 at 7:28 pm PST) by glen

The web server Archibald is being failed over to new hardware. It had been crashy throughout the day with high loads . This shouldn’t take longer than 30 minutes, and it won’t affect email and mysql at all.

Sorry about the trouble.


3:00 pm

Network Update

Posted (September 14th, 2006 at 3:00 pm PST) by micah

After more tweaking and reorganizing today we’re finally seeing some positive results from the router upgrade last night. Load averages everywhere are dropping at the moment but we’re watching and waiting to see how things progress. This is just an update to let you all know that we’re still working on things and that the router upgrade does seem to have helped.

Happy DreamHost Curbed-Optimism Team!


10:54 am

Some Network Issues Remain

Posted (September 14th, 2006 at 10:54 am PST) by Dallas

Our router upgrade seems to have improved performance for some of you but we are still seeing a lot of complaints of little or no improvement at all. Now that we are confident the routers are not presenting bottlenecks performance-wise we will be focusing our attention on the specific areas of the network that are still having problems. We have received a shipment of high-powered ethernet switches and will begin deploying those to problem areas in the network. We are also investigating other options to improve performance.


9:45 am

Automatic update scheduling backed up

Posted (September 14th, 2006 at 9:45 am PST) by andrea

It appears that the automatic update scheduler service wasn’t turned back on last night after finishing with the network upgrades. We turned them back on about half and hour ago, but it may take a while for them to catch up. This affects all new additions and changes made through the control panel, such as adding a domain, changing user password, doing a one-click install, just to name a few.

We sincerely apologize for any inconvenience this may have caused.


 
© 1996-2006, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
11:57 pm

Final Router Upgrade Post

Posted (September 13th, 2006 at 11:57 pm PST) by micah

All up!

There are still some stragglers but at this point 99% of your sites should be up. We still have admins and tech support staff on hand to watch over things and fix problems as they arise. If your site is still down or you’re experiencing some other problem go ahead and submit a support request.

As with the router replacement on Monday, we won’t know any concrete details on the affects of this upgrade for several hours. More information will be forthcoming tomorrow. Thanks for hanging in there with us!

Happy DreamHost No Sleep Yet Team!


11:26 pm

Router Upgrade Update

Posted (September 13th, 2006 at 11:26 pm PST) by micah

While probably 90% of all webservers and mail servers are up, there are quite a few straggling Apaches that are having to be restarted by hand. It looks like they didn’t play nicely with our runlevel changes earlier. There are only one or two on each server but for a couple dozen servers. We’re currently going through all these at the moment so if your site isn’t up yet this might be why.

Once those are cleared out I’ll post again. Your continued patience is appreciated, so far this upgrade has gone quite well!


10:43 pm

Router Upgrade Complete!

Posted (September 13th, 2006 at 10:43 pm PST) by micah

Core networking functionality has been restored. Web and mail servers are on their way up right now if not so already.

We’re working on cleanup of stray services and then we’ll post an “all up” message.


9:04 pm

Router Upgrade

Posted (September 13th, 2006 at 9:04 pm PST) by micah

In addition to the recent router hardware replacement, we will be performing multiple upgrades to our primary core router at 10pm PDT tonight. This will entail the same amount of downtime, roughly 30-60 minutes, to the same services (HTTP, Mail, webpanel, SSH, etc).

For those of you interested, we will be installing 2 2500W power supplies, a Sup720 supervisor, 2 fabric enabled 48 port GigE blades, and a high speed fan tray.

We will post again when the upgrade has been completed and network functionality has been restored.

Happy DreamHost Late Night Router Upgrade Team!


8:47 am

Router Hiccup

Posted (September 13th, 2006 at 8:47 am PST) by micah

It seems our core01 router lost its BGP sessions this morning which caused a very brief networking issue for about half of our network. It already looks like everything is back online already and we’re not sure what happened exactly but we’ll post more information as we have it.


11:45 am

The Network is Still Bad

Posted (September 12th, 2006 at 11:45 am PST) by Josh

Aside from early indications that the router hardware swap was successful last night it seems today that things are depressingly close to the way they were before.

While core01 is running on all new hardware (except power cables and power supply) it was not upgraded. Unfortunately while configuring the new chassis with the Sup720 supervisor and improved blades and extra “high speed” fan tray, we ran into a problem.

The router complained that the power supplies we have (1300W, 27amp) are not “compatible” with the new fan tray. This isn’t to say that they can’t power it, just that somewhere a version or model number is incompatible.

We already knew that we’d need 2500W to power the new supervisor and fan tray, and we’d spoken to cisco techs about running our two 1300W power supplies in cumulative mode for now which basically makes them output 2600W worth of power. They said that should work fine. We spoke to our Cisco network engineer yesterday and he said it should work fine. Cisco’s documentation clearly says:

With insertion of equal-wattage power supplies in cumulative mode:
The system power is the combined power capability of both supplies.

So, it definitely seems that it’s a false limitation somewhere that would be solved by getting two 2500W powersupplies. These are already on their way and scheduled to arrive tomorrow.

The upshot of this is we’ll be having another network maintenance window tomorrow, September 13th, at 10pm PDT.

(Just to get everybody on the same page.. essentially ALL performance and stability problems experienced over the last months have been due to these internal network problems. High web and mail server loads are due to the lag they’re getting when NFS mounting the file system from our file servers. This causes slow/flakey performance across the board. It’s actually not a matter of overloaded servers or anything like that, it’s all this very weird network problem that has had us and Cisco stumped. We’ve now completely replaced all the related physical networking gear and it’s still happening. Cisco support and network consultants have been unable to find any problems with our configuration or hardware and have no explanation as to why things are still behaving like they are. So we’re now upgrading the core routers to be 8 times more powerful.)

Also, we’ve decided to start allowing comments on this blog.. remember the note at the top though: Posting in the comments here WILL NOT help you get your problem resolved faster.

We hope it will at least help foster some better communication related to events we post here.


8:03 am

Poprock webserver crashy (Updated)

Posted (September 12th, 2006 at 8:03 am PST) by andrea

The poprock webserver has been crashy for a day now, and we’ve been trying to bring it back up. However, it looks like the machine may have some bad RAM, so we’re moving it to different hardware. This shouldn’t take longer than 30 minutes, and it won’t affect email and mysql at all.

Sorry about the trouble.

10am PDT – all sites should be operational again. Contact support if you need further assistance.


12:08 am

Router Maintenance Complete

Posted (September 12th, 2006 at 12:08 am PST) by micah

We are confident that all the minor issues following the router hardware swap this evening are now resolved. If you are having any issues with your site or email please contact our Tech Support team.

While initial performance looks promising we’ll need another 12-24hours before we will have concrete evidence that the hardware replacement has resolved the major issues.


10:54 pm

Network Maintenance Complete!

Posted (September 11th, 2006 at 10:54 pm PST) by micah

We have successfully completed the router hardware swap and services are coming back online.

Most webservers and mail servers are already up and running, there might be a few stragglers that we’ll be working on over the next few minutes.

We’ll post another update once everything is completely up.


9:03 pm

Network Maintenance

Posted (September 11th, 2006 at 9:03 pm PST) by micah

The planned network maintenance window for tonight is going ahead in approximately one hour (at 10pm PDT). We will be replacing our entire core router with new hardware during that time.

This maintenance window will entail downtime for all services (HTTP, Mail, SSH, Webpanel, etc…) for approximately 30-60 minutes. We are taking steps to minimize the impact of the afteraffects on our servers and we have Admins standing by in the datacenter as well as extra Tech Support staff available for helping you with any problems you might encounter. We ask you to please wait for a network up post here before writing in to support.

We also recommend you postpone any panel changes until after the maintenance window since updates will likely be delayed.

Your continued patience is very appreciated, we are looking forward to putting this past us as well.

Happy DreamHost Late Night Router Replacement Team!


 
© 1996-2006, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
1:06 pm

Mail server upgrades affecting customers not using DreamHost DNS servers

Posted (September 11th, 2006 at 1:06 pm PST) by jordan

In order to improve mail server performance, IP addresses of our old mail servers are being disabled. This will only affect customers that are not using DreamHost’s DNS servers ( ns1.dreamhost.com,ns2.dreamhost.com,ns3.dreamhost.com) . To check and see if you are going to need the new information, make sure that either your MX and A records for mail are going to a DreamHost DNS name eg. mx1.balanced.randy.dreamhost.com , or are pointing at an ip in the 208.97.132.0/24 network. If you are not sure and need verification please email support and they will be happy to assist you.


10:56 am

Light at the End of the Tunnel

Posted (September 11th, 2006 at 10:56 am PST) by Dallas

The router maintenance scheduled for tonight is essentially a complete replacement of our ailing router as well as upgrades of all of the key components. After several weeks of tracking down the issue we (and Cisco) have finally decided the problem is definitely hardware related. After replacing several of the components of the router over the past few weeks without improvement we will now be replacing the chassis itself (the box everything sits in) as well as upgrading several components. The new router will be several times more powerful than the current one and will be essentially brand-new.

After this maintenance we will be following up with an upgrade of the other router to get it in line performance-wise with this one. We are also discussing further upgrades of the network to add more throughput and allow us to scale ever farther. We will also be switching our core network equipment to a separate power feed with a more reliable UPS system. That process has been underway for a few weeks now.

If we could have done this upgrade sooner, we would have. We were not able to get the replacement parts we need any sooner. The problem seemed like it might be hardware related from very near the beginning of our series of problems but we were not able to convince Cisco support enough for them to provide us with the replacement equipment until now. The problems we have been seeing are very sporadic and difficult to isolate. One possible silver lining in all of this is we are now more experienced with Cisco’s support (this was our first major issue with our equipment) and we know better what we can actually expect from them. We will be reviewing our existing support contracts and investigating possible alternatives that may work better for our needs.


3:07 pm

Network downtime MONDAY night! (UPDATED)

Posted (September 7th, 2006 at 3:07 pm PST) by jeremy

We’re rescheduling the network maintenance previously scheduled for 8pm PDT Sunday night (Sep. 10) to 10pm PDT Monday the 11th so that we can perform a more thorough upgrade. We will be taking the network down for some fairly invasive central network upgrades, so most everything will be down for the duration. We’re now expecting downtime to be in the range of 30-45 minutes due to the extra work, but doing things this way will prevent a second or third outage later in the week. During this time everything (web, ftp, email, panel) will be unavailable.

Happy DreamHost Fans-Of-The-Network Team!


11:14 am

Mail delays

Posted (September 7th, 2006 at 11:14 am PST) by andrea

There is large amount of mail accumulated and waiting to be delivered on one of the machines in your server cluster. This only affects users whose mail is hosted on murdock.dreamhost.com.
The messages are in the process of being delivered. Please allow up to 2 hours for new messages to arrive. This applies also if you’re moving junk messages to your inbox. We sincerely apologize for any inconvenience this may have caused. Please let us know if you have any additional questions.


2:17 am

Baht Buggered (Resolved)

Posted (September 7th, 2006 at 2:17 am PST) by Patrick

Baht went to server heaven. It’s poor components couldn’t take the harsh realities of serving web pages anymore.

It’s being brought back online on new hardware now. Check this space for updates.

-UPDATE (02:56 GMT -07:00)-

Baht has been reincarnated as a shiney new server that has so much more to live for. If you have any further problems with the baht server, please contact support.


6:07 pm

More filer shuffling (All done!)

Posted (September 6th, 2006 at 6:07 pm PST) by DreamHost

Tonight, September 6, 2006, starting on or about 10:00PM Pacific time we will need to move some filers around. This is to increase our overall density per filer. About 5 filers will be moved total covering all clusters. Each one should only be down for about 10 minutes, during which time your email and websites may be inaccessible. You should not lose any email due to the downtime.

We’re sorry for the short notice, I meant to post this yesterday! This is the only night this week we can do it!

–The Happy DreamHost Heavy Lifting Team


3:36 pm

Panel certificate errors

Posted (September 6th, 2006 at 3:36 pm PST) by Ralph

Some of you have gotten errors about the secure certificate for secure.newdream.net being expired. The certificate has been renewed and is installed and working correctly. Certificate errors on the panel should not be happening anymore. Sorry about that.


11:50 pm

Panel Going Down

Posted (September 5th, 2006 at 11:50 pm PST) by Patrick

Parts of the panel are going down for maintenance. Most of it will still be accessible, however changes will take longer than normal. Not to fret though! Any change that you do make to your account via the panel will be queued and executed as soon as we’re done with our maintenance. Check this space for updates.

-UPDATE (01:30 GMT-07:00)-

Maintenance has been completed and everything is back to normal! Everything has been queued and will be updated if you made changes while we were doing maintenance, however if you made a change that absolutely positively must get through or the universe as we know it is going to end, you should make the change again to push it through quicker.


3:09 pm

Natty MySQL Server Restored

Posted (September 5th, 2006 at 3:09 pm PST) by micah

The MySQL server, natty, that blew a raid card this morning and had some other issues has been fully restored now.

The backups were taken at 2:30am PDT so they should be quite recent. If your database on natty isn’t working correctly let us know.

Happy DreamHost Natty Light Team!


10:18 am

Mail delays

Posted (September 5th, 2006 at 10:18 am PST) by andrea

Some of our customers may experience some mail delays this morning. Due to the earlier filer problem (see post right below this one), mail is backed up. The MX machines are pushing through mail again, and it’s slowly catching up. This has affected the filter machines as well, so junk mail delivery is slow too. Things should be back to normal within a couple of hours. No mail was lost.

Sorry about the inconvenience.


 
© 1996-2006, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
8:00 am

Network storage server problems this morning

Posted (September 5th, 2006 at 8:00 am PST) by andrea

One of our network storage servers, aka filer, went down this morning. The admins are working on getting it working again. This is only affecting a very small percentage of our customers (we have a multitude of filers in service), and it means that web and mail servers that are interacting with it are unable to pull files, and hence sites are down or email access is limited. Some servers are (not exclusive list): nash, bundy, alvarado, snicker, and parts of the spunky mail cluster.

We apologize for the inconvenience, and will be posting updates as soon as possible.

Update at 9am PDT: the filer is back up, but let us know if you’re still having problems, and we’ll be happy to investigate. Sites should be showing fine and mail service should resume and catch up soon.


6:33 am

Natty Unhappy

Posted (September 5th, 2006 at 6:33 am PST) by Patrick

One of our MySQL servers, natty, crashed. We are working on it now and it will be back up as soon as possible. If your database is down, chances are good that it’s on natty.

-UPDATE-

The chassis has been replaced, but there is a problem with the drives. We are reconfiguring the RAID array now.

–NEW UPDATE–

The RAID just wasn’t happy, we’re restoring from backup (which was taken at 2:30am PDT this morning) and should have everything back online shortly.


5:17 am

Rexford and Cortes Relocating

Posted (September 5th, 2006 at 5:17 am PST) by Patrick

Rexford and cortes will be going down briefly to be moved to a new rack. Total downtime should be less than 15 minutes. Check this space for updates.

-UPDATE-

Everything is back up and running properly!


4:16 am

Rupee Rocked

Posted (September 4th, 2006 at 4:16 am PST) by Patrick

One of rupee’s partitions has become unreadable. To minimize downtime, the server is being failed over to new hardware (which is new and shiney!).

Check this space for updates.

-UPDATE-

Rupee is back up and running on the latest and greatest hardware!


9:09 pm

MySQL server Elmyra Down – UP AGAIN

Posted (August 30th, 2006 at 9:09 pm PST) by glen

The MySQL server Elmyra has been unresponsive to our attempts at resuscitation . This only affects databases on this 1 specific server. The administrative team is diligently working to get it back up with as little continued downtime as is humanly possible. Updates to follow as efforts progress.

UPDATE Server is UP, a single mysql instance is currently being hammered, it is being worked on to lessen the load.


4:18 pm

Network Update

Posted (August 30th, 2006 at 4:18 pm PST) by micah

After many more incremental improvements and configuration changes we’ve decided to upgrade our core router. The upgrade will increase CPU power as well as total throughput which should definitely alleviate many if not all of the problems we’ve been seeing lately.

The upgrade should be fairly seamless but we will let you know ahead of time if we anticipate downtime.

Hopefully this will help with all the NFS problems that have been affecting site and email slowness.

Happy DreamHost Net-PLEASE-FINALLY-Work Team!


2:40 pm

Comcast forwards to be disabled, and AOL Update

Posted (August 30th, 2006 at 2:40 pm PST) by Karl

Today we have some good news, and bad news!

We’ve been contacted by someone very helpful at AOL and I think we have that problem squared away, at least for now. There have been no further blocks and the AOL contact has indicated that our thresholds for blocking are much higher now. We’re waiting for a bit to see whether this will solve the problem long-term, and are looking at implementing some suggestions they have made in the meantime to ensure that we can hopefully stay on their good side. We still won’t be allowing new AOL forwards or forwards that have been removed to be re-added, but the existing ones won’t be disabled for the time being. I still recommend setting up a local mailbox rather than forwarding any mail if possible, as forwards of any type add a potential failure point in your email’s path.

Now, for the bad news — Comcast has become an increasing problem in the last two weeks and is now completely denying our unblock requests. As a result:

In 7 days, on Wednesday, September 6th, we will be disabling all forwards to @comcast.net addresses.

As a bit of background:

Comcast blocks are atypical from the others that we’ve been having problems with in that they last indefinitely until unblocked manually. Unlike AOL blocks (which phase out automatically after 24 hours – though may reappear) someone has to flip a switch over there for any future mail to go through. The unfortunate part is that they have zero human availability and all we get from their blacklist email address are auto-responses — either the IP is automatically unblocked, or the unblock is denied and the phone number of their abuse/security department is given. Unfortunately, this phone number is a completely unmanned voicemail drop-box. We’ve left no less than six messages on their voicemail in the last couple months, and despite numerous requests we have never received a phone call back or an email response. We’re not even asking that they remove the blacklist — we’re simply asking for more information on why the IPs were blocked, and for a sampling of the typical spam they are supposedly getting from us! In fact, the only response we can get from them (if we get one at all) is an automated form message saying that “most of the email” received from our IPs is spam, which we know, in fact, is false.

Again we regret that this decision had to be made, but we’re currently wasting a great deal of time answering complaints regarding Comcast blacklists, not to mention calling and emailing Comcast to have those blacklists removed (unsucessfully) — and they are a very small fraction of our total email forwards. For reference, our number of Comcast forwards is 1/12 of our GMail forwards, and GMail gives us zero problems while somehow managing to filter mail very well for its user base.

For the people who have Comcast forwards set up, I recommend that you remove them yourself and set up a local DreamHost user where mail can be downloaded via client software or checked via webmail. You can edit the destination of an email address by going to “Mail” -> “Manage Email” on the left-hand side in the DreamHost control panel, and click on “Edit” next to the email address in question. If you have difficulty with this, please contact support.

Next week, for those addresses that don’t have an alternate recipient other than a Comcast forward, we will create a local email account for mail to be routed to, so that nothing is lost.


6:51 pm

Joomla One-Click updated to 1.0.11 – Required Upgrade!

Posted (August 28th, 2006 at 6:51 pm PST) by DreamHost

This is a critical security release, so upgrade your Joomla installs as soon as possible. You can read more about this in the Joomla team’s Release Announcement.

If you used our One Click Installer to install Joomla, you should be able to “one click upgrade” it as well:

https://panel.dreamhost.com/index.cgi?tree=goodies.installer&

Just click on the Upgrade link next to your Joomla installation. Make sure to only do one at a time!


10:35 am

Sake web server

Posted (August 28th, 2006 at 10:35 am PST) by andrea

The hardware on our sake web server is being the replaced. The machine will be up and running shortly. We truly apologize for any inconvenience this may have caused. Please check back here for further updates.


4:20 am

Problems with email authentication

Posted (August 28th, 2006 at 4:20 am PST) by jordan

We are having a few problems with a database that runs our mail authentication for imap/pop/smtp services. This problem is being looked into and should be fixed asap. We apologize for the trouble .


 
© 1996-2006, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
11:47 am

Quick ram swap in a filer (All done!)

Posted (August 27th, 2006 at 11:47 am PST) by DreamHost

Swapping out the ram in a filer. It keeps hanging RSH connections, which is no good as quotas and other stuff can’t update! More to come in the next few days if this doesn’t fix it.


8:40 pm

Moving beverly to new hardware

Posted (August 25th, 2006 at 8:40 pm PST) by DreamHost

Beverly’s network is having some trouble, so I’m moving it onto new hardware.

Update 10:00pm. Beverly is up and working on its new hardware. If you have any sites hosted on Beverly that still aren’t working please contact support.


11:56 am

Brief network downtime again tonight.

Posted (August 24th, 2006 at 11:56 am PST) by jeremy

This time, the network will be going down so we can run some thorough diagnostics on our core routers. I’m expecting the downtime to last perhaps 15-20 minutes, but there’s a chance of some residual effects lasting for a short while afterwards.


10:47 am

Network connection issues still

Posted (August 24th, 2006 at 10:47 am PST) by andrea

The network maintenance last night went about as smoothly as it could have, without any actual downtime to any customers. Unfortunately, the ocassional slowness and downtime due to network connection issues remained for some of our customers. We’ve already opened new support cases with Cisco and our network consultant is getting more information on his.
We apologize for all inconvenience and appreciate your continued patience.


10:40 pm

Network and database maintenance completed.

Posted (August 23rd, 2006 at 10:40 pm PST) by jeremy

Everything with both projects went as smoothly as could be. The network never went down completely, the only symptoms we noticed were sub-optimal routes and network slowdowns as BGP caught back up. All panel and webmail access have been restored as well.


9:41 pm

Network outage reminder

Posted (August 23rd, 2006 at 9:41 pm PST) by jeremy

Don’t forget, in about half an hour (10pm PDT) part of our network will be going down briefly while we replace some faulty equipment. Total downtime should still be in the range of 10-15 minutes.


9:17 pm

Database Back-Up Drive Replacement

Posted (August 23rd, 2006 at 9:17 pm PST) by Patrick

The back-up drive in limbo-mysql is getting replaced now. The server will be down for ~20 minutes. If your database is not responding, it is most likely hosted on limbo-mysql. Check this space for updates.

-UPDATE-

limbo-mysql is up and running with the replaced back-up drive.


4:48 pm

Panel database maintenance tonight

Posted (August 23rd, 2006 at 4:48 pm PST) by DreamHost

We will be performing some maintenance on one of the databases which runs the DreamHost webpanel beginning tonight, August 23, at 10PM PDT (-0700). Ending 1 hour later at August 23, 11PM PDT (-0700). The DreamHost webpanel (http://panel.dreamhost.com) will not be available during this time. Sorry for any inconvenience this causes.

Note: this will -not- have any effect on hosted sites, email nor databases.


4:27 pm

Filer shuffling

Posted (August 23rd, 2006 at 4:27 pm PST) by DreamHost

Tomorrow night, August 24 2006, starting on or about 10:00PM Pacific time we will need to move some filers around. This is to increase our overall density per filer. About 6-10 filers will be moved total covering all clusters. Each one should only be down for about 10 minutes, during which time your email and websites may be inaccessible. You should not lose any email due to the downtime.

–The Happy DreamHost Heavy Lifting Team


3:40 pm

Balrog was K.O.ed

Posted (August 23rd, 2006 at 3:40 pm PST) by Javier

Balrog had a hardware failure and we tried to bring it back up but had no success, We are failing it over right now (moving everyone on it to a new server) and your sites should be back up soon. Sorry about the downtime. We’ll post an update when we have more info

Update: We finished moving Balrog to a
new server now and all accounts on that server should now be up. If you are
having issues with your site please

contact us
.


 
© 1996-2006, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
11:09 am

Dreamhost Hit Counters and Quicktime media streaming downtime.

Posted (August 23rd, 2006 at 11:09 am PST) by jordan

We will be performing upgrades on our Quicktime streaming / Hit Counter services today starting at around 12:00PM PST. The outage should only last about 15-20 minutes. Please note that this will not affect any of your quicktime files on your site unless you have specifically setup streaming support for them in the panel via :

https://panel.dreamhost.com/index.cgi?tree=goodies.streaming&


8:26 am

AOL forwards to be disabled

Posted (August 22nd, 2006 at 8:26 am PST) by Karl

As per AOL’s request of us, all forwards from DreamHost-hosted addresses to AOL accounts will be disabled in one week, on Tuesday, August 29th.

Some background on the problem: We’ve been having intermittent problems with automated AOL volume and spam blocks since February, after our mail system was load-balanced. We’ve also grown a great deal during this time, which has added to the general mail volume passing through our servers and contributed to the problem. The load-balancing served well to even out internal server loads and mail queues, but has not worked out so well for our external relationship with other mail providers like AOL, since we now have more machines connecting on fewer IPs.

To address spam complaints, we set up a feedback loop with AOL that sends any emails generating spam complaints back to us. The servers that send original mail receive generally 0-2 complaints per day, while the MX servers (that receive mail, and only send for forwards/autoresponders/bounces) are getting 10-30+ complaints each. Looking at the individual complaints: almost all of it is due to forwarded mail.

Complaints from forwarded mail happen when someone sends mail to a DreamHost-hosted email address, which in turn is set up to forward mail to an AOL address, and the recipient on AOL’s end clicks “This is Spam”. AOL counts this as a complaint against us, even though the mail just passed through our system and did not originate with us. We’ve been trying to address this by warning and disabling AOL forwards for individual accounts as appropriate, but at this point, this is getting impractical to deal with, and we’re not seeing any less AOL complaints despite the effort (because new ones continue to be added).

We’ve called AOL’s postmaster helpdesk numerous times since February and are only able to reach their first level technicians who ask us to either a) submit another whitelist request, or b) write an email, and then c) call back later. Emails directly to the postmaster and whitelist requests explaining how our mail system works and asking for clarification on their policies regarding forwaded mail have been ignored or replied to with a form email that did not answer the questions asked. Essentially, we’re unable to get an actual human response from whomever is making the decisions over there.

In my most recent conversation with an AOL representative on the phone I was told directly that we should disable AOL forwards, as they will not be whitelisting our IPs while they are generating complaints, and they will not be changing their complaint logic to only target the original spammers.

We regret that we have to make this change and it’s something that we’ve been trying to avoid, but we can’t continue to allow our customers who are trying to send new mail to have problems due to these forwards, and we can’t continue spending copious man hours attempting to manage a problem caused by a single ISP, no matter how large they may be. Finally, AOL’s own representative has asked that we stop forwarding mail to AOL — it can’t get much simpler than that.

For people who have AOL forwards set up, we recommend that you remove them yourself and point them at a non-blocking provider that still offers spam filtering (like GMail) or have it go to a local DreamHost user where it can be downloaded via an email client or checked via webmail. You can edit the destination of an email address by going to “Mail” -> “Manage Email” on the left-hand side in the DreamHost control panel, and click on “Edit” next to the email address in question. If you have difficulty with this, please contact support.

Next week, for those addresses that don’t have an alternate recipient other than an AOL forward, we will create a local email account for mail to be routed to, so that nothing is lost.


3:11 pm

Network hardware replacement on Wednesday!

Posted (August 21st, 2006 at 3:11 pm PST) by jeremy

It’s starting to look like some failing hardware may be the cause of our recent network slowness, so instead of waiting for it to get worse, we’ve managed to get the suspect part RMA’d without verifying the hardware failure. We’ll be doing the hardware swap the night of Wednesday the 23rd at around 10pm. The network will be degraded during this time and a good portion of the network will be completely unavailable. We expect downtime to be no more than 15 minutes and more likely in the range of 5 minutes.

Happy DreamHost Net-Finally-Working Team!


9:10 am

Network issues causing servers to crash

Posted (August 21st, 2006 at 9:10 am PST) by andrea

We are experiencing some networking issues this morning that is causing certain web servers to crash and is slowing down portion of the mail delivery. Our admins are looking into it, and we’ll post an update as soon as we have new information.
The affected servers so far are (a fairly complete list): melrose, patton, ike, ivan, garland, manchester, attila, foothill, deblume and rampart.

Very sorry about the inconvenience.


8:36 pm

Nailer’s sorta … nailed

Posted (August 19th, 2006 at 8:36 pm PST) by Brian@dreamhost

The filer “nailer” crashed just a few minutes ago, and one of our technicians is on his way out to our datacenter, to bring it back to life. It should be back up in about 30 minutes. We apologize for the downtime!

*It looks like it’s taking a little longer than expected. We’re still working on it, and should have things back to normal soon!


2:53 am

Server Moves

Posted (August 18th, 2006 at 2:53 am PST) by Patrick

It turns out that a few web servers need new digs too. Leave it to the MySQL servers to make everyone else jealous!

These servers are being re-racked and will experience some downtime:

flower, twizzler, barranca, krackel, fundip, and dimarco

Check this space for updates.

-UPDATE-

All 6 servers are back online and running properly.


11:21 pm

Venkman being moved onto new hardware

Posted (August 17th, 2006 at 11:21 pm PST) by DreamHost

Webserver venkman failed this evening around 10:10pm pdt, we’re in the process of switching it onto new hardware and hope to have it running shortly.

Update: This was finished around 11:30pm.


10:49 pm

MySQL Server Moves

Posted (August 17th, 2006 at 10:49 pm PST) by Patrick

Two database machines, elmyra and graymalkin, are going down for roughly 20 minutes for a change of location. They’re tired of looking at the same old servers in their current rack and yearn for new company. If your database is down, you are most likely on one of these two MySQL servers. Check this space for updates.

-UPDATE-

Both servers are back up and sipping on margaritas in their new digs.


3:09 pm

Miva Mail Server Changes

Posted (August 16th, 2006 at 3:09 pm PST) by andrea

This message only applies to our customers who have Miva Merchant set up on our servers (any version) and are using the built-in Order Fulfillment Configuration or OpenUI module to send email notifications to the merchant and to their customers.

If you notice that your order confirmations are not going through or are bouncing with a Relay Access Denied error, the following will fix it:

Log into your miva admin and click the Domain Settings link on the left (make sure you’re in Advanced Administration). That will bring up links in the right frame, among them Site Configuration. Click that, and replace your current entry with the word localhost in the Mail Server field. Click the update button and you’re all set. Your order confirmations should start going through again shortly.

Trouble finding where to make the change? Contact support and email us your miva admin login, and we’ll be happy to take care of it for you.


8:14 am

Relay Access Denied

Posted (August 15th, 2006 at 8:14 am PST) by andrea

If you’re receiving Relay Access Denied errors after trying to send mail in your email client, please try the following before you contact support. This applies even if you haven’t touched your settings and it’s been working before:

Make sure that you have SMTP authentication enabled and using the same login that you have set for incoming mail (check your Outgoing Server settings). If that is ok, then your ISP might be blocking port 25, so you will need to change that to 587 (SMTP port.) If that doesn’t help either, then you will need to use your ISP’s SMTP server and login to send mail.

If you’re getting the same error but you’re using Pine to send mail, please add the following to your .pinerc:

smtp-server = mail.yourdomainname.com/user=yoursmtpusername
or leave the default as is, which is blank:
smtp-server =

If after checking on this you’re still getting the above error, let us know. Please provide header and full bounces/errors when contacting us.

Sorry about the inconvenience.


 
© 1996-2006, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
3:53 pm

Power update.

Posted (August 11th, 2006 at 3:53 pm PST) by Josh

Our building has let us know that they’re planning a 2 minute power outage today at 6pm PDT (in about two hours) THAT WILL NOT AFFECT US IN ANY WAY.

(A different feed, a different grid.)

Nonetheless we’ll be here, just because you can’t be too safe.

They also say exactly the same thing for 9pm PDT this Sunday. Again, IT WILL NOT AFFECT US IN ANY WAY.


11:08 am

Server hillmont is having problems.

Posted (August 11th, 2006 at 11:08 am PST) by David

Unfortunately the server hillmont has developed a problem and is currently down. We have an admin looking into the problem now. Stay tuned for more updates. We apologize for the inconvenience.

Update: (1:59pm Pacific) Hillmont had some problems that caused it to be “Failed Over” basically we moved everyone that was on it to a new machine and that new machine is now “hillmont” The problems should now be solved. If you are still experiencing some issues with your site and your on the Hillmont server please
contact us -Javier


4:17 pm

Mail Status

Posted (August 10th, 2006 at 4:17 pm PST) by mir

Good news! Based on testing and feedback from customers it appears we do indeed have the spunky mail cluster properly authenticating at this time. Our head of the administration team also pinned down a network issue that was causing slowdown for webmail users.

One item of note, some Outlook users are still having trouble connecting but that appears to be specific to the client – if this is happening to you please confirm that webmail is working (if so your account is fine but you need to reboot your computer and possible upgrade Outlook to get it synched again).

We are still pushing hard towards making our mail service the reliable juggernaut it once was so please contact us right away at dhstatus@dreamhost.com* if you see a recurrance of persistant email issues (specifically intermittant connection or logging issues, slowdown of specific services like webmail).

*Be sure to write from a proper contact email address listed under your account so that your email gets through!:

https://panel.dreamhost.com/id/?tab=contact


11:09 am

Rails Upgrade Deja Vu!

Posted (August 10th, 2006 at 11:09 am PST) by Terri

It looks like the security hole found yesterday was only partially closed, so we have to do another mandatory Rails upgrade. Hopefully this will not take as long, and also hopefully this will be the last security fix for some time! We will update this post once the upgrade is being installed on servers, and once again when it has been completed.

Update: Packages are built and tested, so we’ll be installing them everywhere. Please email tech support with any problems related to the upgrade.


12:28 pm

Email improvements!

Posted (August 9th, 2006 at 12:28 pm PST) by mir

Just a fast update to let people know what is going on with email:

1. The ongoing network issues are still having some effect but it should have minimal impact on mail (meaning you may see a bit of slowdown or have a connection drop on occasion but nothing persistant due to this).

2. We have added extra servers to the spunky cluster (so if you have users on the machine ‘spork’ you should see an improvement in speed and a reduction in connection errors). We added a total of four yesterday and one today.

3. We just corrected some load issues on looney (mailboxes on smaug) which were causing slowness and connectivity problems. Update: we just activated another server on this cluster and are watching the load to make sure it balances correclty.

Note: we are keeping an eye out for any further issues related to items 2 and 3 as we want mail running as smoothly as it possibly can – we’ll make additional adjustments if needed!

4. We are in the process of upgrading our mail database server – we have it worked out so that this will cause minimal impact on our customers. The only noticeable effect (other than better performance once it is completed) will be delays in updates you make through the panel to mail services and this will be brief (for example setting up a new email address right when we are changing the database that stores your aliases will have to wait until the move is done as you cannot write and move or new data will be lost – the delay should be no more an hour).

5. We have corrected an issue that was causing intermittent login errors. What happend is that the new mail machines we added yesterday weren’t authenticating as expected (it was failing for those using their email address as username rather than their actual mail user). The reason for the intermittent nature of the login success/failures is that when accessing the mail server customers can end up on any of the servers assigned to their mail cluster so and if they happened to reach one of the new ones, login failed. Now all servers will be authenticating properly!

Update for #5: We have all but one server corrected – while we did properly address the failures yesterday there was an underlying problem with one of the packages on the new machines that did not become apparent until we fixed the original problem. Once the last server has been fixed we’ll update the status again.


11:40 am

Mandatory Rails Upgrade

Posted (August 9th, 2006 at 11:40 am PST) by Terri

It seems that there is a security hole in all versions of Ruby on Rails, and so today we’re working on a mandatory upgrade. We will do this as carefully as possible, but it sounds like this needs to be rolled out quickly. We’ll post another update as soon as this is completed.

Update: Packages are built and looking good, so we’re beginning to upgrade everywhere. Please email support if you have any issues with Rails 1.1.5. You can see if your server has been upgraded by running “gem list.”

Update Too! The upgrade is complete across our network. We’ve tested everything out very thoroughly so there shouldn’t be any issues, but please email support if you see any bumps related to the upgrade.


5:32 pm

problems with babyruth

Posted (August 7th, 2006 at 5:32 pm PST) by danwy

Those of you on the babyruth server may have been experiencing some problems today. We’ve found the source of the problem, but unfortunately it requires us moving the server to new hardware. We’re in the process of doing that now and it should be back up and running within the next hour. Sorry for the inconvenience.

UPDATE 6:46pm (PST): All done! If you have any further problems accessing your site, please let support know right away.


6:35 am

Mail sending problems

Posted (August 7th, 2006 at 6:35 am PST) by andrea

Some of you are experiencing mail sending problems this morning. It looks like the database handling outgoing mail authentication is not working properly. The admins are looking into it, and are contemplating changing the db itself, so this sort of sending issue won’t happen any more.

Please check back here for updates and try using webmail to send in the meantime. We apologize for all the inconvenience this has caused.

Update 11:30am PDT – The databases were fixed, so all email sending should be normal again. You may experience initial slowness, but that should go away soon. If you’re still experiencing email sening difficulties, please submit a request and give us a detailed description: we need the email address, the email client you’re using, is this a recent issue and what is the exact error message.

Update 1:20pm PDT – Looks like SMTP was broken again for a while. It’s fixed again, mail is reported to be flowing through again. Very sorry about this.


4:17 pm

Filer and power strip maintenance tonight (Completed!)

Posted (August 2nd, 2006 at 4:17 pm PST) by DreamHost

Tonight we will be performing emergency maintenance on a filer which has started having its watchdog card reboot it every 5 hours or so, and a power strip which can no longer remotely manage.

The filer swap should only take about 15 minutes. There will be very little discernable problems unless your data is actually on there. No data should be lost.

The power strip swap will take about one hour, during which time service will be interupted for these servers:

orangey, rhubarb, socks, putney, bmw, max, calgary, barrance, cortes, twizzler, van, sipke, flower, rexford, krackel, fundip, runts, dimarco

This is all fallout from the recent power troubles. The filer is suspect, the power strip lost its management interface after one of the power blips. We apologize in advance for the downtime, however some preventative maintenance now will help prevent catastrophic failure later!

This went without a hitch. We swapped out the filer and the power strip with minimal breaking of channel nuts. All services listed should be back online. Please write in the technical support if it is not!


1:10 pm

Io Memory Upgrade

Posted (August 2nd, 2006 at 1:10 pm PST) by Patrick

Just like charm, io has lost it’s memory… going down for an upgrade now.

-UPDATE-

Io is back up and working like, uhm, charm.


 
© 1996-2006, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
12:46 pm

Charm Memory Upgrade

Posted (August 2nd, 2006 at 12:46 pm PST) by Patrick

Charm is going down for a memory upgrade. Check this space for updates.

-UPDATE-

Charm is back up in all it’s glory!


1:44 am

grizabella hardware replacement (done!)

Posted (August 2nd, 2006 at 1:44 am PST) by DreamHost

The shared mysql server grizabella lost its backup drive during the power outage it seems. I am going to take it offline in about 15-30 minutes for about 15 minutes to swap out the disk.


8:44 am

Announcement lists stuck

Posted (August 1st, 2006 at 8:44 am PST) by andrea

The annoucement list processor was stuck for a little while.
It is running now, and all the customer newsletters are going through, one by one.

Sorry about the delay!


8:00 am

Scheduled user and domain updates and one-click installations backed up

Posted (August 1st, 2006 at 8:00 am PST) by andrea

Our automated scheduling system fell behind due to our ongoing network problems.
We have them in queue, but it may take anywhere from 2-10 hrs for all of them to run.
We are planning on adding another machine just to process the backlog, so it may be a shorter wait.

We apologize for the inconvenience this has caused.

Update at 12:30pm PDT: the backlog was reduced, and according to the latest estimation the latency is down to 95 minutes.
We appreciate your patience and continued understanding.


4:13 am

Quick HD swap in limbo-mysql shared database server (done)

Posted (August 1st, 2006 at 4:13 am PST) by DreamHost

The shared high-usage limbo-mysql server needs to go down for a new backup drive. The old one has failed. This will affect a very small number of our customers, and should only take about 15 minutes.

The server is back online!


11:16 pm

Mail authentication (un)broken

Posted (July 31st, 2006 at 11:16 pm PST) by DreamHost

It seems one of the Database servers that our mail system uses as its backend isn’t properly replicating right now. We are in the process of restoring that, which will fix the mail authentication problems. No email is being lost! Hopefully all will be better in 1 hour.

Everything seems OK now, email is flowing freely, I can seem to check and send my own personal email. Please write in to support if you are having any more troubles!


3:28 pm

More network troubles…

Posted (July 31st, 2006 at 3:28 pm PST) by Josh

The spanning tree protocol of one of our edge switches seems to be breaking portions of our public network.. we’re on the case trying to correct the configuration and restore service across our system as quickly as possible.

:(

Update

After disabling spanning tree on several of our distribution switches this has been resolved. We are still seeing strange network issues that we will continue to work on fixing.


1:50 pm

Panel / Webmail back up.

Posted (July 31st, 2006 at 1:50 pm PST) by Josh

We’ve fixed the problems with the errant dist switch and things should be back to where they were this morning.

There is still a more general (not as severe) network-wide problem we’ve been working on for the last week. We still have two tickets open with Cisco and have an expert networking consultant on the scene. We’re cautiously optimistic we’ll have that resolved as well within the next day and be all the wiser for it.

Our apologies again for this long string of bad.


1:00 pm

Panel / Webmail is down

Posted (July 31st, 2006 at 1:00 pm PST) by Ralph

While tracking the internal routing issues we’ve been experiencing, problems with one of our distribution switches caused the panel and webmail to go down. We are working on bringing them up now. We’re very sorry for the inconvenience… and all the problems we’ve been having the last three weeks..


12:10 pm

WordPress one-click install upgraded to 2.0.4

Posted (July 31st, 2006 at 12:10 pm PST) by DreamHost

Lots of bug and security fixes, so an upgrade is recommended. Read more the new version at wordpress.org


 
© 1996-2006, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
1:18 am

Nilla Nailed Again

Posted (July 31st, 2006 at 1:18 am PST) by Patrick

Unfortunately nilla’s problems are not quite yet over. It is being worked on now. Check this space for updates.

-UPDATE-

Nilla is back up and happy again!


1:50 am

Nilla hardware failure

Posted (July 30th, 2006 at 1:50 am PST) by Terri

The memory in Nilla has failed this morning. All users on Nilla are right now being switched over to new hardware. The new machine should be up and running within 30 minutes. Sorry for the inconvenience.

Update Nilla is back up and running again, smooth as silk! Thank you all for your patience.


3:16 am

Dreambooks Are Back, Jack!

Posted (July 29th, 2006 at 3:16 am PST) by Javier

Ok we have brought the Dreambook service back online and it should be ok now. If you are still having problems, contact support and let us know your Dreambook address so we can check it out.


11:00 pm

Post Power Outage

Posted (July 28th, 2006 at 11:00 pm PST) by Patrick

Services are back up and running system wide. There are a few straggler Apaches that need a little extra attention but those are being whipped into shape now. We are no longer running off of the generator and are back on city power.

If you are still unable to connect to your site/server/ftp/jabber please contact support.


5:58 pm

Webmail back up!

Posted (July 28th, 2006 at 5:58 pm PST) by Ralph

Webmail is now back up and working correctly. We were able to repair the database rather than restore it so everything should be great.


4:30 pm

Webmail connection errors

Posted (July 28th, 2006 at 4:30 pm PST) by Ralph

We are having problems with one of our central databases. This was caused by the power outtage that we experienced this afternoon. Our Admins are currently looking into this matter. Everything will be up and running shortly. Again, we sincerely apologize for any inconvenience this may have caused.

UPDATE: Webmail is still currently having problems and we may need to restore the database from backups if we can not fix it soon. Sorry about the continued problems.


3:05 pm

Brief Explanation for today’s power outage

Posted (July 28th, 2006 at 3:05 pm PST) by Dallas

This note came from our building just now:

The Garland Building experienced a dead short which resulted in a
brief power outage today, July 28, 2006. The air conditioning,
elevators, and the electrical utility have all been restored.

While on generator power, a dead short occurred from one of our
internal telecom users. We are investigating where the dead short
occurred. A follow-up memo will be sent by the end of the business
day reconfirming our transfer at 11:30pm tonight. We are currently
on DWP power until further notice.


2:34 pm

Power restored

Posted (July 28th, 2006 at 2:34 pm PST) by Ralph

Power has been restored and most servers are up again. The few servers that are still down are being rebooted by our administrators. Still waiting on further information from the datacenter. Our apologies for the downtime this caused.


1:46 pm

Power outage at the datacenter

Posted (July 28th, 2006 at 1:46 pm PST) by Ralph

There has apparently been another power outage at the datacenter. We’re waiting for more information and we’ll post here as soon as we hear anything.


11:54 am

Upgrading Charm’s RAM (Postponed)

Posted (July 28th, 2006 at 11:54 am PST) by DreamHost

In a few minutes Charm will be going down for a quick ram upgrade. Total downtime should be about 5 minutes.

It looks like charm had less ram slots in it than we though. We do not have large enough sticks of ram on hand to perform the upgrade. I will order the correct ram overnight from our vendor and we will perform the upgrade Monday or Tuesday.


 
© 1996-2006, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
10:38 pm

Router Upgrades Complete!

Posted (July 27th, 2006 at 10:38 pm PST) by micah

We’ve completed the software upgrades on our two core routers. Everything looks to be up and running normally. We’ll have more information soon on whether this has resolved the recent latency issues.

Happy DreamHost Net-Working Team!


9:36 pm

Emergency Router Upgrades

Posted (July 27th, 2006 at 9:36 pm PST) by micah

In an effort to stabilize our internal network issues (causing most of the sluggishness and high loads lately) we are performing OS upgrades on our two core routers this evening. The upgrades will begin in 30 minutes at 10pm PDT and shouldn’t last for longer than 5 minutes each.

This will result in our network being completely unavailable for at least a portion of that time. This page will be updated immediately following the upgrade to keep you abreast of our network status.

Your continued patience as we work to eradicate this last hurdle is much appreciated.


9:50 pm

Failing over bob

Posted (July 26th, 2006 at 9:50 pm PST) by DreamHost

Bob is having some system disk problems and is in the process of being moved to new hardware right now. We’re working on it as fast as possible and will post an update when we’re done.

Update 10:05pm: bob’s up and running on new hardware, if you’ve got any sites hosted on bob that aren’t working now please let us know.


1:20 pm

Internal Network Update

Posted (July 26th, 2006 at 1:20 pm PST) by Dallas

The root cause of the slow website access and general sluggish performance of our servers has been isolated to one of our core routers. The router is not doing its job properly and we are in communication with Cisco about it. We may end up performing emergency maintenance to restore full functionality but that’s not known at this time.

We know now this problem has been ongoing in a sporadic way for a couple of weeks but we had not identified it completely. It then became a lot more severe in the last few days and has caused the ongoing issues many of you have experienced.


3:28 pm

continuing intermittent problems

Posted (July 25th, 2006 at 3:28 pm PST) by danwy

Many of you have been writing in to say that you’re still having problems with your site speed and email connections. We’re still experiencing intermittent difficulties with NFS traffic and high loads on our web and mail servers as a result of the recent filer and power problems that we’ve documented here. Our admin team is continuing to work on these problems and we’re hoping to get everything back to normal soon. We thank you all for your patience during these trying times.


11:35 am

Dreambooks are down.

Posted (July 25th, 2006 at 11:35 am PST) by Javier

Currently the server that runs the Dreambook service is down. We have an Admin assigned to it and he will have it back up as soon as he can. Keep an eye on this page for an update on the issue. We apologies for the downtime


10:59 am

Catch-all Autoresponders Disabled & Spamcop Update

Posted (July 24th, 2006 at 10:59 am PST) by Karl

Catch-all autoresponders are being disabled right now, per the announcement on the 10th:

http://www.dreamhoststatus.com/2006/07/10/catch-all-auto-responders-to-be-disabled/

We ended up giving an extra week on this one, but we didn’t receive any volume of emails from folks this time around saying that they really depended on catch-all autoresponders, so we don’t expect any real problems. If you’re caught by surprise by this change, please read the prior announcement for the reasoning behind the change, and contact us afterward if you have questions.

SPAMCOP UPDATE:

As of our most recent Spamcop report at 10:22 AM (Pacific) we had 0 IPs listed in Spamcop, so this should not currently be a problem. However, this is an ongoing intermittent issue. When a block pops up, that IP will be blocked for 24 hours at which point it will be de-listed automatically. We cannot speed this up.

As a refresher – certain RBLs like Spamcop have ‘spam traps’ set up that particularly prey on autoresponders and other potential forms of backscatter. We do not believe our recent issues with Spamcop to be caused by spam originating on our servers — but rather primarily due to Spamcop’s policies on autoresponders/backscatter. This change, as well as some other network-based changes that are in the works, are aimed at directly improving the amount of autoresponses to illegitimate email (that tends to auto-respond to an innocent third-party who had their email address spoofed by a spammer) and eventually backscatter as a whole. These are not simple changes to make as they affect potentially thousands of customers and our mail network’s architecture, and we thank you for your continued patience on this matter while we make steady steps toward resolving the issue as a whole.

As per usual, Webmail can be used to send on a different set of IPs, which should get through to Spamcop-protected addresses in the meantime, when a block is in effect. For those of you who have ISPs that will allow you to relay email as your domain, you may wish to try using their SMTP servers, temporarily.


2:46 am

Lingering Email Issues Resolved

Posted (July 24th, 2006 at 2:46 am PST) by Dallas

We have identified some lingering email issues related to some equipment coming up with incorrect configurations after the power outage. We believe all of these issues have now been resolved. We’ll also be reviewing the configuration setup of the equipment to make sure it can all handle loss of power more gracefully in the future.

If you are still experiencing problems sending or retreiving email, let our support team know right away. We have a backlog of support requests at the moment but we will get to them all as soon as we can.


1:55 am

Melrose server being replaced

Posted (July 24th, 2006 at 1:55 am PST) by Justin@DH

The melrose server had a network card go bad, we are failing it over to new hardware right now.

Update: The new server is up and the configuration scripts have finished.


3:09 pm

Power Update

Posted (July 23rd, 2006 at 3:09 pm PST) by micah

One of our datacenters is taking us offline in a few hours to repair a broken Power Distribution Unit. Fortunately there’s only a few live servers in there so we’re simply going to move them over to our main datacenter. This will only affect the following servers: popcorn, rollo, compton, a shared database server and our whois server.

Downtime should only be approximately 15-20 minutes. We appreciate your continued patience during this string of issues.

Update: All of these servers have been moved and are now up. Total downtime was 5-10 minutes. No other servers will be affected by our secondary datacenter going offline.


 
© 1996-2006, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
3:21 am

All Services Back Up

Posted (July 23rd, 2006 at 3:21 am PST) by Dallas

At this time it looks like we’ve gotten everything back up. All websites, email and all other services should be up and running. If you are still experiencing problems with one of your services, contact our support team right away.


10:28 pm

Severe Power Outage

Posted (July 22nd, 2006 at 10:28 pm PST) by micah

The power outage has been confirmed and is ongoing. Evidently DWP suffered a power plant loss in Northern California (due to record temperature levels) that quickly triggered outages in a few Los Angeles neighboorhoods. Our datacenter is currently under generator power and we working on getting everything back online as quickly as possible. Once we hear more about DWP’s status we’ll make another post.


6:30 pm

Possible power outage

Posted (July 22nd, 2006 at 6:30 pm PST) by jeremy

At least a portion of our datacenter seems to have lost power very briefly. We have technicians driving to the datacenter now to see exactly what the situation is. If your site is down, we should have it taken care of shortly.


10:23 am

Cheeky downtime

Posted (July 22nd, 2006 at 10:23 am PST) by Justin@DH

The cheeky server had a issue with one of its network cards and is being failed over to new hardware. Apaches are already starting to come online and the failover should be done shortly.


11:53 pm

Sekhmet databases back online

Posted (July 19th, 2006 at 11:53 pm PST) by DreamHost

All the databases from sekhmet are back online. The process took longer than expected, and we ended up having to just move all of the data to a new server. Everything looks to be online with it though, and queries are running again in force.


2:55 pm

An update on the recent stability issues

Posted (July 19th, 2006 at 2:55 pm PST) by danwy

We are still in the process of offloading data from the filer that caused the problems over the last couple of days (we started this shortly after it started failing). There is a LOT of data on that filer, and it is going to take a while to move it all off. The filer keeps crashing, so somebody is here 24 hours a day to restart it, swap out drives and continue offloading where it left off. Through this process, we’ve learned a lot of new tricks to help recognize these types of filer problems before they happen, so we shouldn’t experience problems of this magnitude in the future.

Unfortunately, this hasn’t been the only problem. The filer crash started a chain reaction that heavily affected other parts of our network, saturating their network interfaces (which is the cause of the slowness/outages some of you are still seeing). We have worked around some of these issues by splitting up switches to have less traffic being pushed through them. We’re also in the process of auditing our entire network to ensure that the NFS traffic won’t be affected like this again in the case of a hardware outage. There have been quite a few things that have caused the problems of the past few days and our admins are busy trying to get all of them ironed out. We have a lot of work still ahead of us to get things back to normal. We are making a lot of progress though, and once things are back to normal things should be a lot more stable than they previously have been. We thank you for your patience as we try to get things back to normal here.


1:56 pm

sekhmet mysql server problems

Posted (July 19th, 2006 at 1:56 pm PST) by danwy

Some of you with databases located on the sekhmet server may have noticed problems with it over the last few hours. It had multiple drives fail on it and we’re in the process of replacing them and will be bringing the server back up shortly. We apologize for any inconvenience this causes you.

UPDATE 5:30pm (PST): We’re still trying to get this back up. It looks like multiple drives in this server have failed and will not fully rebuild. We’re going to try a few more things and if it’s unsuccessful we will be moving it over to new hardware.

UPDATE 6:41pm (PST): We have managed to get the machine back online in a read-only mode and are copying all the data off of it to a new server. This process we’re hoping will be done in the next hour or two, so that your databases will be back online.


1:08 am

Filer problems causing outages

Posted (July 19th, 2006 at 1:08 am PST) by jordan

We are currently having problems with the filer we have been working to fix since yesterday. It has crashed again and we are in the process of bringing it back up. This outage may be affecting http/ftp/email service for your account. We apologize for the outage and hope to have everything restored to normal asap.


11:29 am

Still cleaning up after the filer problems

Posted (July 18th, 2006 at 11:29 am PST) by danwy

Most of the filer problems from yesterday have been resolved, but there are still a few outstanding issues. One of our switches is saturated at the moment and is causing high loads on some servers. If you’re still experiencing slow load times with your site or email, this is probably the cause. Our team of admins (who haven’t got much sleep the last couple of days) are down in the datacenter now trying to remedy this situation and hope to have it back to normal shortly. We thank you all for your patience as we try to clean up the last of these issues.

UPDATE – 3:45pm (PST): We are still working on the problems with the switch that are causing some of our customers to experience delays and outages with your sites and email. The problem was with a switch for one of our racks not being able to keep up with the traffic being pushed through it. We just added a second switch to the rack and pushed it live and it looks like the loads on all the affected servers are dropping pretty steadily. We have a team of admins continuing to monitor the situation and to clean up any remaining issues. Hopefully with this second switch in place that will leave this string of unfortunate events behind us.


6:36 am

Server Compton has been restored

Posted (July 18th, 2006 at 6:36 am PST) by jordan

All services have been restored for customers on the server compton. The issue was related to networking issues we were having earlier. If you are still having any troubles getting to your site hosted on this server please let us know.

Thanks!


 
© 1996-2006, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
3:55 am

Server Compton Outage

Posted (July 18th, 2006 at 3:55 am PST) by jordan

We are currently having a few problems with the server compton. If you site is located on this server you may be experiencing service outages effecting ftp/web services. The problem is being worked on and should be fixed asap.


7:01 pm

Network problems resolved.

Posted (July 17th, 2006 at 7:01 pm PST) by jeremy

Today’s network problems were caused by not 1, not 2, not even 3, but 4 separate issues.

First, one of our older distribution switches decided to give up on life.
After replacing it with newer hardware, everything ran just fine for a couple hours. Then, it decided our network topology had changed (apparently due to a different default configuration since the old days when it was deployed), and started dropping a good percentage of traffic.
This issue also affected another switch that some of our file servers plug into, so customer data was unavailable to a good portion of our hosting servers.
Consequently, one of our more active mail servers tried to dump all of the backed up email all at once, saturating some of the network that it’s on.
Once these configuration issues were resolved, one of the network interface cards in our main firewall machine died, requiring a quick swap.

We’re still working on some of the after-effects, and there may be some slowness while things get caught up, but everything should be back to normal later tonight.


5:03 pm

The problems are worse..

Posted (July 17th, 2006 at 5:03 pm PST) by Josh

It seems like the problems are more widespread with our network than just our internal stuff.

We’re actively working on it, and apologize for any possible downtime this may be causing customer services.

We will update here as soon as we know more!


3:42 pm

Panel and Wiki down..

Posted (July 17th, 2006 at 3:42 pm PST) by Josh

We’re having some problems right now with our firewall for public websites we have that connect to our private internal databases.. we hope to get it fixed soon, but in the meantime, https://panel.dreamhost.com/ and http://wiki.dreamhost.com/ (and http://mailboxes.anydomainwehost.com/) are down.

We’re doing our best to get it all better ASAP!


1:17 pm

Filer Offload

Posted (July 17th, 2006 at 1:17 pm PST) by micah

After several attempts at getting this weekend’s misbehaving filer to recover we’re finally offloading the data to a new filer that was created yesterday. The offloads will take a while but we’ll be closely babysitting the old one for the duration.

Our apologies again for all of the downtime and other issues related to this bad filer, we’ve been working as much and as fast as possible to keep the effects to a minimum.

—–

Due to the other problems we had today (http://www.dreamhoststatus.com/2006/07/17/network-problems-resolved/) the offload of the file server just began.


10:43 am

Filer Update

Posted (July 17th, 2006 at 10:43 am PST) by Terri

Right now our admin team is deploying a new filer to replace the one that is still having issues. This should take no longer than an hour, although copying the data from the limping filer to the new one will take over 48 hours.

We’re very sorry for the inconvenience this has caused anyone. Our admin team is working as quickly as possible to get the new filer hardware up and running.

We will update once again when the new filer is deployed.


4:00 am

More Filer Issues!

Posted (July 16th, 2006 at 4:00 am PST) by Brian@dreamhost

Well, we thought we were out of the woods, however, the same filer went down again. Our administration team is heading over to our data center now, and should be performing the steps to recover from the crash. We’re doing everything we can to make sure downtime is at a minimum, and apologize for the trouble!

Update

The filer is back online after having lost both of it’s spare disks. At this point all of the shelves, cables, and associated hardware has been replaced and yet this filer is still killing disks. We’re doing everything we can to avoid data loss but backing up your email and site data is strongly encouraged. There will be more information to follow as it becomes available.


12:14 am

File Server Finale

Posted (July 16th, 2006 at 12:14 am PST) by micah

We’re confident that the problematic file server is stable now so we’re putting all web servers back online. The configuration changes are in the process of going live so everyone’s sites should be up shortly. If you continue to have problems after a few hours from now then please let support know.

Happy DreamHost Fingers-Crossed Team!


5:07 pm

File Server Issues (cont’d)

Posted (July 15th, 2006 at 5:07 pm PST) by micah

After fighting with the file server all night we’ve finally got it back up and in a usable state. No data was lost during this outage.

At the moment we’re slowly bringing just mail access back online so we can keep an eye on things and make sure everything is running smoothly. If all goes well then we’ll turn website and shell access on too. While the situation has vastly improved, things are still not 100% so your continued patience is appreciated. Check back here for further details later.

Happy DreamHost All-nighter Team!


2:09 am

Network fileserver troubles

Posted (July 15th, 2006 at 2:09 am PST) by DreamHost

One of our fileservers is having some serious problems, and is causing several of our shared hosting servers to become unresponsive. We’re working to swap out the filer now and get everything back online.

Watch this space for updates.

We think we have the problem narrowed down now and fixed. The fileserver has been running a consistency check on the volume for about the last hour now, and should be done in about 30 more minutes. Once this is done we can get service restored!

It looks like the file system check wasn’t as finished as I thought it was. It will be several more hours until it finishes. We’re very sorry about the downtime, we are working as fast as we can.

Update 3:50pm pdt: We still have no eta one when this will be resolved. However, we recently issued some fixes to the affected mail servers that should allow queued mail for those not directly affected to start coming in.


 
© 1996-2006, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
5:40 pm

Soto down

Posted (July 14th, 2006 at 5:40 pm PST) by DreamHost

We’ve got an admin working on it right now and hope to have it up shortly.

Soto is up! We did a security upgrade on it and a couple other servers then forgot to make sure soto came back up afterwords like the others. Sorry about that!


11:45 am

Avalon problems

Posted (July 14th, 2006 at 11:45 am PST) by DreamHost

This morning Avalon has been fairly crashy. We are actively looking into the problems. More to come!


5:37 pm

Discussion List Upgrade

Posted (July 13th, 2006 at 5:37 pm PST) by Terri

Discussion lists for many customers will be down briefly Monday, July 17 2006, at around 11 a.m. PDT. During this brief downtime, which we anticipate should not go over an hour, we will be upgrading the discussion list software. No discussion list email will be lost during this process.

We apologize if this causes any inconvenience, but this is all part of our effort to make sure our customers have access to the most stable, secure services possible!

Update: Due to filer issues this upgrade has been cancelled until further notice.


12:09 pm

Panel Images

Posted (July 13th, 2006 at 12:09 pm PST) by micah

Images on our webpanel are currently broken do to a server that’s being upgraded. They should be back up and viewable soon!

No other panel functionality is affected by this upgrade.

Update

The upgrade is complete and images are working again!


3:31 am

Rollo Rolled Up!

Posted (July 13th, 2006 at 3:31 am PST) by Patrick

Rollo took a nose dive around 3:30am (GMT -700). It is being failed over to new hardware now and will be back up and running soon. Check this space for updates.

–UPDATE–

Rollo is back up and happy. If you are having having problems with your hosting on rollo still, please contact support


4:13 pm

Catch-all Auto-Responders to be disabled

Posted (July 10th, 2006 at 4:13 pm PST) by Karl

It’s established that catch-all email addresses are big ol’ spam magnets. With one of these active, you’re going to get spam, and very likely lots of it. Last month we disabled forwarding of catch-alls (in most circumstances) to external addresses because this puts our servers in the position of forwarding a whole lot of spam, which not only was putting strain on our servers, but was also resulting in spam complaints (and blocks) for email that didn’t even originate within our system.

We’re taking another step in tightening up the liability of catch-alls and have disabled the setup of new auto-responders for catch-alls, starting now. Existing catch-all auto-responders are still active, but we will be disabling those next Tuesday, July 18th. {An email is going out to those who have these set up… err.. now!}

The reasons for this are fairly simple: spammers often send to hundreds of random addresses at a single domain. With an auto-responder active for a catch-all, each of these will send an email back to the sender. What’s worse is that *almost always* the “From” field on a spam email is a spoofed address at an innocent domain. The owner of that spoofed address then receives a truckload of spam in the form of auto-responses, which they in turn submit complaints for to their ISP or RBLs such as Spamcop, which — you guessed it! — results in our servers getting blocked, or our abuse department getting flooded with complaints.

You will, of course, still be able to set up auto-responders for individual addresses. Given that the general purpose of an auto-responder is to toss back a temporary vacation message, or a “we’ll get back to you soon” acknowledgement in the case of a support desk, these sorts of things are better off with individual, specific email addresses anyway!


4:07 pm

MediaWiki Updated to 1.7.1, now requires PHP5

Posted (July 10th, 2006 at 4:07 pm PST) by DreamHost

Make note of the new PHP5 requirement before you upgrade if you’re running PHP4 on your domain. Full release notes available here: link.


9:42 pm

Cienega unhappy

Posted (July 8th, 2006 at 9:42 pm PST) by DreamHost

cienega.dreamhost.com is currently not being happy. We’re currently working to make it happy again and hope to have it online again soon.

Update Sat Jul 8 22:20:35 PDT 2006: cienega back online.


1:17 pm

Recent DNS Update Problems Fixed

Posted (July 8th, 2006 at 1:17 pm PST) by Justin@DH

There was an bug in the dns script Friday that caused some dns record changes not to add, update or remove as expected. We have corrected the issue and dns updates are functioning normally again. If you have waited the normal couple hours for dns changes to propagate and are still not seeing the records on our nameservers, please submit a support ticket.


10:42 am

Database issues

Posted (July 8th, 2006 at 10:42 am PST) by Javier

The MySQL database mimsey has gone down and we have an Admin in route to get it backup again, It should be up soon as he just a couple of miles to travel. Sorry for the downtime. We’ll keep you posted.

Update Sat Jul 8 11:18:23 PDT 2006: mimsey is back online, all databases on it are intact.


 
© 1996-2006, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
3:41 pm

Venice going down for memory upgrade

Posted (July 5th, 2006 at 3:41 pm PST) by DreamHost

The server venice.dreamhost.com will be going offline in 10 minutes (thats Wed Jul 5 15:50:00 PDT 2006) to have some memory added to it. Total downtime should only be 10 minutes. This will only affect websites hosted on this server, so if you’re not sure if you’re on venice, chances are, you are not. Email for accounts on venice will remain online during the upgrade.

Update: venice back online


3:09 pm

Charm going offline for a memory upgrade.

Posted (July 5th, 2006 at 3:09 pm PST) by DreamHost

The server charm.dreamhost.com will be going offline in 10 minutes (thats Wed Jul 5 15:20:00 PDT 2006) to have some memory added to it. Total downtime should only be 10 minutes. This will only affect websites hosted on this server, so if you’re not sure if you’re on charm, chances are, you are not. Email for accounts on charm will remain online during the upgrade.

Update Wed Jul 5 15:35:41 PDT 2006: charm is back online.


10:11 am

Email from webservers suspended

Posted (July 1st, 2006 at 10:11 am PST) by DreamHost

We have had to temporarily suspend outgoing email from our webservers due to an enormous spam run. We are having trouble pinning down the exact cause, however it is a distributed spam run across multiple servers (at least 20 that we have found so far). Once this has been resolved we will reenable it and queued email should go out.

This does not affect regular email like you send through your email client or webmail.

Update Sun Jul 2 01:00:00 PDT 2006: Email service for webservers has resumed. Queued email on the webservers is being sent and appears to be caught up now.


12:09 pm

Panel Hanging

Posted (June 30th, 2006 at 12:09 pm PST) by mir

We have corrected this – there was a problem with a specific database that it was trying to access besides the central one. After fixing that access was restored so there shouldn’t be any more problem related to this issue.


12:21 am

Webmail problems resolved

Posted (June 30th, 2006 at 12:21 am PST) by DreamHost

We have located the cause of the webmail database troubles, and believe we have fixed it. Webmail should not have these connection problems anymore.


12:19 am

Hardware upgrade complete

Posted (June 30th, 2006 at 12:19 am PST) by DreamHost

The upgrades to our central management hardware went smoothly and everything is back online. Queued updates are running and they should all be caught up after a couple hours.


2:09 pm

Centinela being moved to new hardware

Posted (June 29th, 2006 at 2:09 pm PST) by danwy

As many of you that are hosted on the centinela server may have noticed, there have been quite a few problems with it over the past couple of weeks. We have been investigating the problems since they began and have not been able to find any software related causes. At this point, we believe the problem may be hardware related. In hopes of fixing this server for good, we are going to be moving it over to new hardware within the next hour. You may notice a little downtime for your web based services (web, ftp, etc), but email should be unaffected. Hopefully on this new hardware, centinela will be much happier.


11:23 am

Hardware upgrade Thursday, June 29 2006

Posted (June 28th, 2006 at 11:23 am PST) by DreamHost

Tomorrow night, June 29 2006, we need to perform an upgrade on some of our central management hardware. This will involve stopping configuration updates. The window is from 9:30pm Pacific time through 12:00am, however the expected downtime will be around 60 minutes. All changes made through our web panel will be stored and catch up in the few hours after our new hardware is live.

If you need to make any changes, make sure to get them in by 9:00pm!


1:42 am

Bubs went bonkers!

Posted (June 28th, 2006 at 1:42 am PST) by Patrick

One of our database servers, bubs, is in the midst of a hardware failure. There is an admin in the datacenter now re-arranging hardware and it will be back online soon. If your database is not responding right now, there is a good chance you’re on bubs. Check this space for updates.

Update Wed Jun 28 02:21:56 PDT 2006: Back online, everything is cool. Any further database troubles are likely not the same problem.


4:24 pm

Minor Analog Report Changes

Posted (June 27th, 2006 at 4:24 pm PST) by Dallas

In the next few days the default report you see when viewing your stats reports (your_domain.com/stats) will switch to be the monthly summary (updated daily) rather than the daily report. The monthly report is the format used by other popular web statistics software packages and is probably the most useful report for most people.

Additionally we will follow up on this later by turning off the daily report altogether. Before we do that you will have the option to set it to stay on for your domains if you like to use it. You’ll also be able to turn it on for domains in the future if you would like. Turning off the daily statistics for people who do not use it will save precious cpu resources on our shared hosting servers.


 
© 1996-2006, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
12:49 pm

Joomla One-click Install updated to 1.0.10

Posted (June 26th, 2006 at 12:49 pm PST) by DreamHost

All existing Joomla! users MUST UPGRADE to this version, due to several High Level vulnerabilities that affect ALL Previous versions of Joomla! For more info: http://joomla.org/content/view/1510/74


11:40 am

Last chance to donate!

Posted (June 26th, 2006 at 11:40 am PST) by Josh

This is just a note that tomorrow we’ll be switching the charities at our panel’s Charity section.

So far we’ve got 49 donations to Heifer International totaling $929.50, and 34 donations to The Open Source Initiative, totaling $604.21 .. c’mon people, let’s up that a bit!

Remember, we’ll be matching the totals…..


8:47 am

WebMail again

Posted (June 25th, 2006 at 8:47 am PST) by micah

The webmail MySQL database ran out of connections again this morning. I’ve just restarted it so webmail is working again. I’ll keep an on it to see if I can’t catch it before it gets too bad again.

One of the problems we’re working on is that when it runs out of connections it does so very very quickly so it’s hard to catch. We’ll get something in place to either notice this issue more quickly or prevent it all together.

Thanks for your patience, sorry about the repeated inaccessibility.


4:23 am

Webmail down – now up

Posted (June 24th, 2006 at 4:23 am PST) by DreamHost

Webmail is currently down, we’re working on getting it fixed, and hope to have it working again shortly.

Update: June 24, 4:27am: And webmail is working again, sorry about that


11:52 pm

Panel Slowness

Posted (June 22nd, 2006 at 11:52 pm PST) by DreamHost

The large number of extra updates running today caused panel slowness due to slowness on one of the central database machines. Many of the updates are now complete and the panel is back to normal. Please contact support if are experiencing any difficulties. Apologies for a very unfortunate day.


5:21 pm

DNS Updates Running

Posted (June 22nd, 2006 at 5:21 pm PST) by Ralph

If you sent in a message with the category of “site down” and with your domain in the relevant domain field then updates have completed on your domain. This is for support message received before 5:00 pm -0700 GMT. The updates have been going well and are fixing your sites. If you wrote in before 5 pm and you are still having problems please keep in mind that it may be your cache that is still bringing up the domain in a broken state. You can test to see if your site is working by having someone not in your network check your domain, or you can log onto your server and type: (note: you need shell access to do this)

lynx domain.com

Replace domain.com with your domain. If your domain comes up then it is fixed and you are just viewing the cached domain in your web browser. If your domain is still broken please contact support again with “site down” and your domain in the relevant domain field.


11:16 am

DNS Issues cont.

Posted (June 22nd, 2006 at 11:16 am PST) by Ralph

If you are still having problems viewing your domain please submit a support ticket with “Site Down” as your category. We are currently setting up groups of updates for customers writing in with “Site Down” as the category for your support request.

https://panel.dreamhost.com/index.cgi?tree=support.msg&

Category: Site Down

This should allow us to update your domain faster.

****UPDATE****

Please also include the domain you are having problems with in the relevant domain and URL field. That’ll help us get to the domain even faster.

****UPDATE 2****

Another issue we noticed is that people are submitting subdomains for the site down and domain report. DNS is based on the top level domain, so just let us know the top level domain you are having problems with if it is a sub domain. So example.com instead of sub.example.com. Thanks.


11:10 am

Quick filer swap (all done!^2)

Posted (June 22nd, 2006 at 11:10 am PST) by DreamHost

One of our filers is giving errors right now so we need to do a quick swap before it becomes a major problem. This will affect your websites only if you are on this filer. Downtime is estimated under 10 minutes. There will be no dataloss, the disks are fine and we can do a clean shutdown. We will post an update when this is completed.

Everything went as planned, total downtime was about 2 minutes. If you are still having problems please let support know!

It looks like it’s still crashing so the filer itself wasn’t the problem. Going downstairs to swap more hardware and see if that solves it. Again, downtime should be less than 10 minutes.

We have now swapped all of the hardware in the filer except the physical disks themselves. We are hoping this will resolve the problem.


3:11 pm

DNS Issues

Posted (June 21st, 2006 at 3:11 pm PST) by Javier

We had some errant updates to our apache webservers which resulted in a few domains receiving the wrong IP. We are updating those now with the correct IP which should be completed within the next two hours. With the maximum time to live on DNS of two hours that means everything should be working within 2-4 hours at most.

Update Thu Jun 22 01:43:55 PDT 2006: The time estimates above have shown to be inaccurate. The updates are taking much longer to run than first expected. However, we have taken some steps to speed up the remaining ones as much as we can. We are expecting the updates to finish sometime late this morning.


10:33 pm

OS Upgrades for Shared Database Servers

Posted (June 20th, 2006 at 10:33 pm PST) by DreamHost

Beginning June 21, at 1AM PDT we will be upgrading the OS on our shared-hosting database servers to the latest version. The actual downtime for your databases should only be a couple of seconds, and thus should not be noticeable, but this is just a heads up.

It will likely take a couple of weeks to get them all upgraded, with the number of servers upgraded each night being low at first and slowly ramping up.

UPDATE Thu Jun 22 02:14:20 PDT 2006: Note, this is only updates of the Operation System on the servers. The MySQL version will not change. Sorry if this caused any confusion.


 
© 1996-2006, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
4:16 pm

Webmail MySQL Database problems

Posted (June 20th, 2006 at 4:16 pm PST) by micah

The disk on our Webmail MySQL server filled up earlier today which caused the database to become unavailable. Downtime was intermittent and probably totalled around 25 minutes. We’re freeing up some space and limiting the amount of logs we keep around so that this does not happen again. Sorry for the inconvenience!


2:57 pm

Data move mixup affecting about 1500 users (Fixed!)

Posted (June 20th, 2006 at 2:57 pm PST) by DreamHost

We are in the process of fixing this up now. Things should hopefully be back to normal in about an hour or two. That number also includes email-only users, however it should not be affecting mail delivery.

This should all be fixed now. Please write to support if you are having trouble logging in, or you have data missing.


10:54 am

Some incoming email delays on the spunky cluster

Posted (June 20th, 2006 at 10:54 am PST) by Dallas

We are seeing incoming email delays for customers in our spunky cluster. We believe the delays are a result of a much higher than normal incoming email load, possibly due to spammers targeting our servers. We are looking into the situation and will have it resolved as quickly as possible.


11:09 am

Catch-all forwards disabled

Posted (June 19th, 2006 at 11:09 am PST) by Karl

In just a few minutes we will be disabling catch-all addresses that forward to external addresses that are not to GMail, per the previous posts on this subject:

http://www.dreamhoststatus.com/2006/06/07/catch-all-email-addresses-may-no-longer-forward-to-outside-mailboxes/
http://www.dreamhoststatus.com/2006/06/12/69/

If you have any trouble with your catch-all forwards to GMail or domains whose mail is hosted here at DreamHost, please contact our support department!


6:10 am

Genki downtime

Posted (June 19th, 2006 at 6:10 am PST) by Terri

One of genki’s network cards has failed, and we are failing genki over to new hardware. Genki should be back up within an hour at most.


2:15 am

Temporary DNS slowdown

Posted (June 18th, 2006 at 2:15 am PST) by DreamHost

One of our authoritative DNS servers was just having some troubles which slowed its response time, causing some DNS timeouts. The problem is resolved now, and queries are returning to normal as it begins to cache its data again and should be back upto full speed soon.


7:28 pm

Panel login issues resolved

Posted (June 16th, 2006 at 7:28 pm PST) by pete

The panel issue that was affecting some users where a blank page was being served upon login is now resolved. If you still notice any unusual behavior with our panel, please submit a ticket to support.


2:46 pm

DNS Issues

Posted (June 16th, 2006 at 2:46 pm PST) by Javier

We have a DNS server that has gone down which is causing mail not to work for some users. The Admin team is looking into this right now and should have it resolved shortly :-(


11:16 pm

Bandini getting moved

Posted (June 15th, 2006 at 11:16 pm PST) by DreamHost

Bandini has been having some strange network issues. We’re moving it to new hardware now. Everything should be up and running in about 30 minutes.


1:40 pm

Joomla updated to 1.0.9

Posted (June 15th, 2006 at 1:40 pm PST) by DreamHost

Our one-click install of Joomla has been updated to version 1.0.9. A bunch of security vulnerabilities and bugfixes are included with this version, so you should upgrade immediately through our control panel.


 
© 1996-2006, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
1:24 pm

Switch upgrades this week and next.

Posted (June 15th, 2006 at 1:24 pm PST) by jeremy

To better support some new features we’re implementing, we’ll have to upgrade the OS on some of our switching hardware. We’ve already tested the upgrade on a limited basis without any issues, so we don’t expect any during this run of upgrades. Total downtime for your sites during the upgrade will be approximately 2-5 minutes and all upgrades will be done between the hours of 9pm and 11pm PDT.


10:44 pm

Panel Login Issues

Posted (June 14th, 2006 at 10:44 pm PST) by DreamHost

We’re aware of issues logging into the panel, and are working to fix them. A workaround that has succeeded for some is to click the link https://panel.dreamhost.com/index.cgi once you’ve logged in and gotten a blank page.


4:58 pm

Incoming email servers no longer allow outbound email

Posted (June 14th, 2006 at 4:58 pm PST) by Dallas

Our email servers that handle all incoming email from the Internet will no longer allow the sending of outbound email. If your DNS is hosted by us, this will NOT affect you at all. The incoming email servers are set up to only handle incoming email and not outgoing email already. Previously they could still be used to send outgoing email and this was exploited by some spammers and phishers to get around some of our network security measures.

If you host your DNS elsewhere AND if your DNS is not set up correctly for the email servers for your domain, you may begin having trouble sending out email through our servers. If that is the case for you, contact our support team for assistance.


1:31 pm

WebCalendar One-Click Install upgraded to 1.0.4

Posted (June 13th, 2006 at 1:31 pm PST) by pete

Our version of WebCalendar in the One-Click Install panel has been upgraded to 1.0.4. If you are currently running WebCalendar 1.0.3 or lower, you should upgrade immediately as security vulnerabilities have been discovered in those older versions.


3:21 am

Trackback spam on gradius squelched

Posted (June 13th, 2006 at 3:21 am PST) by DreamHost

It looks like there was some heavy trackback spamming going on against the server gradius. This was causing it to crash. I am identifying source IP’s and adding them to the firewall as they stick out. The server should stabilize now.


12:58 am

Minor goof on DB server Sizzle! (Fixed!)

Posted (June 13th, 2006 at 12:58 am PST) by DreamHost

It looks like the MySQL data directory wasn’t properly put onto the big partition on that server. I am moving it over now, however in order to get a consistent move I have to stop all MySQL services on there. It should only take a few minutes for the move.

All better!


4:54 pm

Bandini has crashed

Posted (June 12th, 2006 at 4:54 pm PST) by danwy

Some of you on the bandini server may have noticed that is is unresponsive as of about 4:30pm (PST). We rebooted it, but it is having kernel problems upon coming back up. We are in the process of moving it over to new hardware and it should be back up and running shortly. Sorry for the inconvenience!


11:55 am

Catch-all Forward Disabling Postponed, Dedicated and GMail Exempted

Posted (June 12th, 2006 at 11:55 am PST) by Karl

We appreciate the feedback/concern that has been sent in via support over the weekend regarding the catch-all forwarding change. I’m sorry if we caused anyone to panic! We are simply eager to resolve the blacklisting problems we’ve had as quickly as possible.

We’re going to postpone the actual disabling of external catch-all forwarding until next Monday (June 19th) to give some of you who need to coordinate with your own clients more time to do so. You still can’t add new ones in the panel, but existing catch-all forwards will not be disabled until the 19th.

Further, since it seems a *lot* of people are forwarding catch-alls to GMail to use their fancy filtering, and the fact that Google hasn’t been causing us problems with blocks, we’re going to exempt forwards to GMail from the disabling. It is not feasible to offer this to all domains except those that we know are blocking us regularly (such as AOL) because many email providers employ realtime blacklists (RBLs: http://en.wikipedia.org/wiki/DNSBL) that do spam checks, and some of these tend to ‘ding’ us for forwarded spam as well, and contribute to mail getting blocked in a whole slew of mail systems around the internet.

So, less-wordy summary:

-Catch-all forwarding will be disabled next Monday, June 19th.
-You can still forward catch-alls to GMail accounts.
-[Edit] Almost forgot: Dedicated Servers will not be affected, even though they got the initial notices.

As usual, thank you for your feedback and patience.


4:16 pm

Larchmont’s recent downtime!

Posted (June 11th, 2006 at 4:16 pm PST) by Brian@dreamhost

Today, larchmont was the victim of a Denial of Service attack. We’ve firewalled the IP address sending the attack, and reconfigured the IP addresses that were effected. This resulted in about 30 minutes of downtime. Sorry about that! =-)

Happy DreamHost-we’re-tired-of-DDoSes-lately-team!


2:17 am

Brief DNS outage

Posted (June 11th, 2006 at 2:17 am PST) by micah

It seems our primary DNS server, ns1.dreamhost.com, had stopped responding to DNS lookups for about 20 minutes just now. This caused intermittent issues with people’s sites and email.

Restarting the DNS server seemed to fix the issue but if you continue to have problems accessing your sites or email please let us know.


 
© 1996-2006, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
2:23 pm

WordPress 2.0.3, phpBB 2.0.21, and 50+ themes for WordPress!

Posted (June 9th, 2006 at 2:23 pm PST) by Josh

The title says it all!

Our one-click installation of WordPress has been upgraded to 2.0.3 .. AND we now automatically install over FIFTY totally-sweet themes with each one-click installation (or upgrade).

We also upgraded phpBB to 2.0.21 which came out a couple of days ago!

Oh yeah, and MediaWiki 1.6.7 tooooooo!

Install or upgrade from our One-Click Installs area!


8:48 am

mysql load (resolved!)

Posted (June 9th, 2006 at 8:48 am PST) by brians

We are experiencing mysql load issues. We believe that it’s being caused by a bad switch. Our Admins are in the process of replacing the switch. Sorry for any inconvenience this may have caused.


The switch has been switched! This should hopefully resolve any poor network response time to/from the affected mysql servers.


3:53 pm

Outbound mail from webservers

Posted (June 8th, 2006 at 3:53 pm PST) by pete

Some users have experienced an issue where mail sent from their web server or website has been stuck — no failure, but no delivery. We tracked the problem down to our outbound policy server, which has now been corrected. All mail should now be processed. Any mail that you did not get a bounce message about will be delivered in the next couple hours as our policy server works triple time to get all the queued up mail taken care of ASAP.


9:37 am

Webftp issues resolved

Posted (June 8th, 2006 at 9:37 am PST) by andrea

The webftp issues you may have experienced this morning are now resolved. You should be able to log into webftp through the panel.

Again, sorry about the inconvenience!


8:56 am

WebFTP Issues

Posted (June 8th, 2006 at 8:56 am PST) by andrea

Since earlier this morning, we’ve been experiencing intermittent webftp issues. Not all customers are affected. If you are, please try to use alternate methods for accessing your web files. You can try the regular FTP connection or use a FTP client. For the latter you can find more information in our Wiki:

http://wiki.dreamhost.com/index.php/FTP

The problem is being looked into and we’ll be updating this page as soon as we know further. We apologize for the inconvenience this has caused.


12:14 am

Formosa Failover

Posted (June 8th, 2006 at 12:14 am PST) by Patrick

Formosa’s hardware died around midnight. It is being failed over to new hardware now and will be back up and running soon! Check this space for updates.

-UPDATE-

Formosa is back up and running smoothly on new hardware! If you are experiencing any problems that were not occurring before the failover, please let support know!


9:34 am

Catch-all email addresses may no longer forward to outside mailboxes.

Posted (June 7th, 2006 at 9:34 am PST) by Josh

A lot of the reason we’re getting blocked contantly from sending email to providers like aol, comcast, etc.. it turns out is just due to forwarding-only email addresses on our system.

What happens is, people set up an email address @theirdomain hosted with us, and set it to forward to say, their aol email address. Then, they get spam @theirdomain. It forwards to aol. Aol sees it “originating” from us (when we’re just forwarding it on), and block us as spammers.

There’s not TOO much we can do about this problem, as we have to allow people to set up forwarding-only email addresses!

However, the vast majority of the spam we forward on looks like it’s from people who have catch-alls @theirdomain.

Already a while ago we stopped allowing catch-alls to forward to aol.com email addresses.. and now we’re taking it a step further and not allowing catch-alls to forward to ANY outside email address.

We’re sorry we have to do this. Catch-alls are very useful tools, but unfortunately they’re HUGE spam magnets.

Starting right now, you can no longer create catch-alls that point to anything other than a regular DreamHost mailbox.

This Friday (June 9th) we’ll be turning existing catch-alls forwarding to outside email addresses off. (We’ll notify anybody with a catch-all set up like this via email separately as well).

We’re sorry about this drastic change!


1:46 pm

Blacklist Status

Posted (June 5th, 2006 at 1:46 pm PST) by Karl

Bad news: Looks like both our main mail IP and our new-ish IP that we’re using to test blacklist solutions were listed in Spamcop this morning. We’re looking into it!

Good news: The Comcast/GoDaddy blocks, we believe, were due to them simply not liking the hostname of one of our IPs. They would bounce messages back saying either “residential IP” or “dynamic IP”, both of which were clearly false. We had this *same exact* problem with another IP back in February, and simply renaming the IP fixed it with both providers. We did this yesterday and it has definitely propagated today. I don’t see any new Comcast problems in our queue currently. I do see a couple secureserver.net bounces, but they also include a message regarding Spamcop — so there’s the caveat to that.

If you experience new incidences of either Comcast or GoDaddy bouncing mail (and there’s a fair chance of this happening with secureserver.net), please send those to support with the full bounce error and a timestampped header. You only need to do this once!

We are working on permanent resolutions for these blacklisting problems. The best thing you can do to assist us in the meantime is to NOT report spam you receive on external email accounts that your Dreamhost domains forward to. Doing so may potentially ding Dreamhost for the complaint, even if the mail did not originate on our servers. If you’re hosting mail forwards for other people, tell them too! Certain ISPs (such as AOL) scan every hop in the headers when screening mail, and complaints count against every IP before it reaches them.


2:04 pm

More outages (Updated!^2)

Posted (June 3rd, 2006 at 2:04 pm PST) by DreamHost

We’re experiencing what looks to be another attack, and are working to deal with it right now. We’ll post an update when things are resolved.

The attack has been dealt with and the cleanup is almost done.


Just letting the Happy DreamHost Customers know that we’re stilling working on this as fast as we can. The dos left a lot of our servers in very strange states due to network problems. We are rebooting them as fast as we can. We are on the upswing as far as we can tell!

Almost there, ivan isn’t rebooting right so one of our technicians is on his way to the site to give it a stern talking to. Sorry this is taking so long.


6:21 pm

The fun never ceases!

Posted (June 2nd, 2006 at 6:21 pm PST) by jeremy

Twice today, while battling all the other issues, we came under Denial of Service attacks (and big ones at that!). Each one was about 900 megabits a second, and affected a whole range of services. You may have noticed your sites being slow, mail being slow or not responding at all, and possibly even a crashing server here and there. With the exception of the 1 web server that was actually being targeted, everything should be completely back to normal now.


 
© 1996-2006, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
11:43 am

Configuration issues fixed!

Posted (June 2nd, 2006 at 11:43 am PST) by jeremy

There were a couple of issues working together to create the backlog of updates this morning, both of which have now been found and taken care of. Judging from the current rate at which we’re catching up, we’d expect most updates to be complete within an hour or two, but some may take slightly longer.


10:47 am

Wierdo configuration issues

Posted (June 2nd, 2006 at 10:47 am PST) by DreamHost

Changes in the control panel are currently not going through to our live servers. It appears to be an issue with our configuration servers. All changes are currently stored and there is no need to make them again as far as we can tell.

This has been happening since last evening. We are working on figuring out why. No ETA :(


11:31 am

Mail problems and mail problems fixed..

Posted (May 31st, 2006 at 11:31 am PST) by DreamHost

About an hour and a half ago mail started failing for pretty much ALL customers..

After some research, it seemed like dns was the problem. However, things seemed fine on the dns servers themselves. Something was wonky with the network it looked like.

Poking around for a while finally revealed the problem, the load balancer our mail machines were behind was out of (65,000) ports! Everytime our mail servers send mail they have to do a dns lookup, and to do so they needed to go out through the load balancer to our dns servers, which would then use up a random port waiting for the return packets. Now that we’ve got nearly all our mail servers behind the load balancers though, it was apparently using up all the ports concurrently, causing time-outs and failures.

To fix this, we set up another dns server just for mail and put it behind the load balancer! Things are working again now and this problem shouldn’t rear its ugly head anymore.

Our apologies for this.. hopefully this is close to the last wrinkle to iron out with our new load balanced email setup.


5:50 pm

Spamcop Update

Posted (May 30th, 2006 at 5:50 pm PST) by Karl

We received a de-listing confirmation on our main load-balanced mail IP a couple hours ago and are keeping an eye on the reports in the meantime. Some people may still be getting bounces, but this should lessen within the next several hours. If this crops up again, please contact support unless we’ve acknowledged it here first.


9:30 am

Un-Shanked by .Mac!

Posted (May 30th, 2006 at 9:30 am PST) by Karl

We received confirmation of de-listing from .Mac’s blacklist this morning (a courtesy few provide, that we certainly appreciate!) and they have agreed to whitelist this IP so that it no longer triggers their volume blocks alone.

If you experience further blocks to @mac.com addresses after about 9 AM Pacific this morning, please send them to our abuse address and we’ll take a look right away.


7:39 am

Shanked by Spamcop

Posted (May 30th, 2006 at 7:39 am PST) by Karl

It looks like Spamcop blocked our main mail IP early this morning. We’re currently working on resolving this, and generally Spamcop blocks should be lifted much faster (and should not be as recurring) as problems such as the ongoing AOL issue (which they have set up to trigger automatically, on a daily basis, which we are also working on!).

In the meantime, webmail should still get to Spamcop-protected addresses, so I’d recommend using that if you need to send mail that is urgent.


3:11 am

Dead NIC on Fig

Posted (May 30th, 2006 at 3:11 am PST) by Patrick

One of Fig’s network card’s died earlier. The machine is being failed over to new hardware now and will be back up and running to the world soon. Check this space for an update!

-UPDATE-

Fig is on new hardware and back up and running!


11:13 pm

Arlene back online

Posted (May 29th, 2006 at 11:13 pm PST) by DreamHost

Ok, the database server “arlene” has had its data restored to the last backup, which was from the early morning hours of today (May 29), PDT (-0700). Changes made to the affected database since then were lost, unfortunately. We will be investigating the root causes of the hardware failures, and what needs to be done to our system to prevent further occurences of it.


5:16 pm

Mysql Server Arlene Outage

Posted (May 29th, 2006 at 5:16 pm PST) by jordan

We are currently having problems with one of our shared mysql machines “arlene” , this may be causing outages for mysql based websites. We apologize for the trouble and hope to have it back up and running asap.

Update 9:33PM PDT The server’s main harddrives failed, and we are having to restore the databases from backup. We’re not certain how long this could take, but we will post here when it is finished.


2:07 pm

Beta-Testers Wanted!

Posted (May 29th, 2006 at 2:07 pm PST) by DreamHost

For a long, long time we’ve used the Suggestions section of the Web Panel to get ideas from you, our loyal and attractive customers, as to what kind of stuff you’d like us to work on.

Since we made that Suggestions section, the far-and-away most popular suggestion has been to allow email logins based on your email address instead of some crazy m4834584854857 username.

Well, we’ve done exactly that! The backend has secretly been live ALL WEEKEND! Before we can consider the job done, however, we need a bunch of people to test it out, shake it down, and tell us if it breaks.

So…if you’re a super-big-nerd (like me) and feel like adjusting your email client’s settings, try using your email address everywhere you currently have a username.

Keep in mind that this might suddenly stop working, and you’ll have to change your client’s configuration again!

You can post good/bad feedback to this part of our discussion forums:
Discussion Forum!


 
© 1996-2006, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
8:09 pm

Hops not so hoppy.

Posted (May 28th, 2006 at 8:09 pm PST) by DreamHost

Our shared hosting server “hops.dreamhost.com” is having troubles with one of its network cards, and as such is down. We’re in the process of moving it to new hardware now.

Update: hops.dreamhost.com is back online on its new hardware.


5:04 pm

soda and fidget going down for some quick ram upgrades

Posted (May 25th, 2006 at 5:04 pm PST) by DreamHost

The servers soda.dreamhost.com, and fidget.dreamhost.com, will be offline for a couple minutes each while we stuff some more ram into them. This will only affect websites for those of you who are hosted on these servers, email will remain online.


5:55 pm

Some Lingering Email Issues

Posted (May 24th, 2006 at 5:55 pm PST) by Dallas

We have identified some lingering email configuration issues affecting some users (about 1-2%). The common symptom is ‘Relay Access Denied’ errors resulting when someone tries to send you email. We are now correcting the configurations and everything should be back to normal in the next hour or so (as of 5:40 PM PDT). If you are experiencing these issues on your own domain you can force a configuration update sooner by going to the Manage Email section of our web panel and adding a new address. You can remove it again afterwards. That change will trigger the needed update.


1:30 pm

SMTP Authentication Continued

Posted (May 24th, 2006 at 1:30 pm PST) by Karl

Another cluster began exhibiting the same symptoms as the other cluster that was previously fixed. This should now be fixed as well. Tests on both clusters indicate mail should be authenticating and processing in the mail queues.


11:43 am

Swarthy SMTP Authentication Update

Posted (May 24th, 2006 at 11:43 am PST) by Karl

It looks like this should now be fixed. The admins pinpointed and resolved a problem with the mail load balancing. Mail is now being authenticated, though due to the number of people now trying to send out mail, the mailqueues are a bit hefty. The mail is processing and being sent though, and we’re keeping an eye on it just in case anything goes awry.

Again, thank you to all affected for your patience during this time.


10:53 am

SMTP Authentication Issues Again — Swarthy Cluster

Posted (May 24th, 2006 at 10:53 am PST) by Karl

We’ve been having SMTP Authentication issues for much of the morning and are working on diagnosing and resolving the problem as soon as possible. In the meantime POP/IMAP email should work just fine, as should sending and receiving via Webmail.

Again, we apologize for this iconvenience and thank you for your patience as we continue to address this problem.


2:19 pm

Mail sending and delivery issues – Swarthy cluster

Posted (May 23rd, 2006 at 2:19 pm PST) by andrea

As of earlier today we have been experiencing mail sending and delivery problems on some of our mail servers, caused by database corruption. The admin team is investigating and is working on a resolution. The database is being repaired, but it takes a while to go through all affected domains.

Your mail is affected if you noticed some or all of the following symptoms: Email appeared to be sent but wasn’t getting delivered and incoming mail was bouncing with Relay Access errors up until 1PM PDT, when we stopped mail sending from and delivery to the affected servers. All affected mail should be queueing on the sender server until we enable mail delivery again.

We apologize for all inconvenience this has caused and will post an update as soon as possible.


10:51 am

Mail Authenticaion Issues Resolved – Spunky cluster

Posted (May 23rd, 2006 at 10:51 am PST) by micah

There was a problem with the password file on several of our mail servers this morning. We’ve corrected the issue and are looking into ways to monitor this in the future. We’ve also been working on rolling out authentication via MySQL database for the past month. We should have that completed shortly so that this type of thing doesn’t happen again.


10:30 am

Mail Authentication Issues – Spunky cluster

Posted (May 23rd, 2006 at 10:30 am PST) by micah

Several of our mail servers (almost 2 dozen) are experiencing periodic authentication errors this morning. They are also extremely loaded.

We’re not sure what is causing these problems but we’re investigating as quickly as possible. Your patience regarding this matter is appreciated. More information will follow.


11:17 am

Junkmail Filter Delays (updated!^2)

Posted (May 22nd, 2006 at 11:17 am PST) by Dallas

We are seeing some congestion on our junkmail filter servers. Some email is being delayed as a result. Most of the delays should be short (less than 10 minutes) but some may be longer (a couple of hours or more). We are going to be migrating the back-end junkmail database to a more powerful server today as well as adding another front-end filtering machine. We expect these changes to resolve the delays.

We’re doing tests on the new server right now, and in doing so we had to restart the database backend. Email is going to be delayed about 15 minutes thats coming through the junkmail filter while we get some new settings in place to make everything newer-better-faster!


Mail bound to our junkmail filtering servers has been flowing again for awhile now, no more changes tonight.


 
© 1996-2006, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
12:06 pm

MySQL server ‘sarabi’ having problems

Posted (May 21st, 2006 at 12:06 pm PST) by danwy

As some of you may have noticed, the MySQL server ‘sarabi’ is having some problems. The server has crashed and is not responding to multiple reboot requests. One of our admins is on their way to the datacenter to manually bring this server back up and restore its functionality. It should be back to normal shortly.


9:01 am

Strange mail looping issue

Posted (May 20th, 2006 at 9:01 am PST) by DreamHost

There is a strange mail looping issue on one of our mail clusters. I believe I have the problem nailed down, however I haved turned off postfix on our inbound MX servers. This will only affect a limited number of customers. No email is lost, and it is only looping for a single mailbox of a single customer, but it’s causing mail queues that are growing at a rate of hundreds per second. All email should be flowing inbound again in about 15 minutes.


8:26 pm

AOL blocking Dreamhost emails again

Posted (May 19th, 2006 at 8:26 pm PST) by DreamHost

AOL is currently blocking emails sent from some Dreamhost mailservers. We are working to get the block removed. This happens a lot, and no one is more frustrated about the situation then us. We have complied with everything AOL has asked us to do to prevent this from happening and they continue to block us regularly.

Currently it seems forwards are the problem, so if you have a forward on one of your Dreamhost email addresses to an AOL email address please consider removing it or at least enabling spam filtering.

If you find that emails you send through us to AOL are getting blocked you can write support with the full headers of the blocked email.


1:23 pm

Ruby problems

Posted (May 19th, 2006 at 1:23 pm PST) by jordan

Last night we had a problem with our ruby on rails package causing the ruby gems to be unavailable. The problem has been fixed as of 10:30 GMT -0700 . If you are still experiencing problems using ruby please contact support.


12:08 am

Continued Reports of Spunky Mail Password Problems

Posted (May 19th, 2006 at 12:08 am PST) by Ralph

The problem with mail reported yesterday unfortunately caused a problem in some mail clients. There have been numerous reports that problems still occured after we updated that the problem was fixed. Some email clients still repeatedly ask for your password and bring up the dialog box. Some users have reported that simply rebooting their computer has fixed the problem. Another solution that is currently working for customers is to delete your current setup, restart your email client, then set up your account again. We are very sorry for this inconvenience. If these solutions do not work for you, please contact support and we’ll be happy to help.


12:17 pm

Mail sending problems last night

Posted (May 18th, 2006 at 12:17 pm PST) by DreamHost

Last night email sending was broken for a few hours due to an internal server problem. The server which handles throttling of email wasn’t properly running. As soon as the problem was noticed we began working on finding the cause and getting mail up and running again. We identified the problem at around 5am and had a working solution in place just after 6am. We’re sorry for the inconvenience, in the hurry to get things working we didn’t remember to update this status page.


4:11 pm

Spunky mail problems fixed

Posted (May 17th, 2006 at 4:11 pm PST) by DreamHost

The problem reported earlier about Corrupted Email Configuration is now resolved. All addresses appear to be working correctly now. If you have any current mail problems please contact support and we’ll be happy to help. The final configuration finished about 30 minutes ago and all current mailbox tests are successful. (16:00 GMT -0700)


11:55 am

Corrupted Email Configuration

Posted (May 17th, 2006 at 11:55 am PST) by DreamHost

Unfortunately an email configuration file became corrupted. The symptoms were password request dialog boxes repeatedly coming up or not being able to send or receive messages. This was on the spunky mail cluster. Only some users on the spunky cluster were affected. Users on other clusters were not affected. We are currently reconfiguring mail service on spunky. We apologize for the inconvenience this has caused to the affected users. (12:00 pm GMT -0700)


9:46 am

Pocky downtime

Posted (May 17th, 2006 at 9:46 am PST) by micah

The server pocky became completely unresponsive this morning. I went down to investigate after a remote powercycle didn’t bring it back up. It appears that the boot record became corrupted somehow and 10 minutes with a rescue disk didn’t seem to help. It is currently being failed over to new hardware and should be functioning normally soon.

If your sites/users are on Pocky this would have affected HTTP, HTTPS, FTP, SSH, and Jabber.


11:27 pm

Central Database Maintenance Completed

Posted (May 9th, 2006 at 11:27 pm PST) by DreamHost

All services are now back online!


 
© 1996-2006, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
2:43 pm

Junkmail Delays Fixed

Posted (May 9th, 2006 at 2:43 pm PST) by jordan

The problem with mail delays have been fixed. Delays were due to our blacklisting server going down. A backup blacklisting server has already been added to correct the problem for good.


10:24 am

Junkmail Delays

Posted (May 9th, 2006 at 10:24 am PST) by micah

We’re seeing some minor email backups on our junkmail servers so if you’re using our junkmail filters your mail might be a bit delayed.

We’re working on this issue as quickly as possible.


10:42 pm

Central Database Maintenence

Posted (May 8th, 2006 at 10:42 pm PST) by DreamHost

Tomorrow night (Tuesday, May 9) at 10pm we will be taking several of our central databases offline for some maintenence. This downtime will affect the DreamHost Panel and customer webmail. Things should be back online within 2 hours. This will -not- have any effect on customer sites. Your email will continue to be sent and received, and for those of you who use a POP or IMAP, those mail clients will work normally, only webmail will be inaccessible during this time.

Summary details:
When: Tomorrow night, Tuesday May 9, 10:00pm-11:59pm Pacific (-0800 GMT)
What: DreamHost Panel and customer webmail.


10:08 am

Ivan Server Downtime

Posted (May 6th, 2006 at 10:08 am PST) by micah

The server Ivan ran out of memory earlier this morning and crashed. All attempts at remotely rebooting or powercycling failed and there is a tech en route to the datacenter now. Ivan should be back up and running again shortly.


3:09 pm

Dreambook database maintenence

Posted (May 5th, 2006 at 3:09 pm PST) by DreamHost

Just a quick note that Dreambooks will be offline for a bit while we move its database to a different machine.

Ok, Dreambooks are back online. The actual move hasn’t been done yet, but its back online until further notice, sorry for the false info.


2:12 pm

Archibald update

Posted (May 4th, 2006 at 2:12 pm PST) by Terri

We found the cause of the attack and have gotten it under control. The server is online and the service under attack is updating. We apologize for the inconvenience.


1:36 pm

Archibald is being attacked… Were working on it

Posted (May 4th, 2006 at 1:36 pm PST) by Javier

currently the archibald server is being syn flooded, we are working on getting this fixed.


9:42 am

New Domain WHOIS Privacy Feature

Posted (May 2nd, 2006 at 9:42 am PST) by DreamHost

As you also might have seen in the recent newsletter, we’re now offering a WHOIS privacy feature, totally for free!

This means that when you register a new domain, transfer a domain to us, or modify your WHOIS info, you’ll be given the option to use our privacy feature instead of your personal contact information.

The purpose of this is of course to protect your privacy from scammers, spammers, and crazies! You’ll even get a forwarding-only email address set up at yourdomain.com@proxy.dreamhost.com that delivers all email sent to it to your real mailbox. This way you can still get important domain-related emails, but can easily filter them if you’d like.

Mmmmm, enjoy!


9:38 am

No more Miva, no more osCommerce, more Zen Cart!

Posted (May 2nd, 2006 at 9:38 am PST) by DreamHost

As you might have seen in the newsletter, we have stopped offering both the Miva e-commerce system (it was free on Code Monster and Strictly Business) and the osCommerce one-click install.

The reasons for this were many-fold, but mostly we felt that the open source package Zen Cart was a better alternative. It’s actually based on osCommerce, but seems to have a much more active development community. So yep, that’s it.. you can now install Zen Cart where osCommerce used to be at our one-click installs area!


9:34 am

Minor Change to Privacy Policy

Posted (May 2nd, 2006 at 9:34 am PST) by DreamHost

This is just a note to mention we’ve added the phrase (except in cases where required to by law by law enforcement agencies) to our privacy policy.

We felt it was important to make this clear, especially in light of our new Domain WHOIS Privacy feature. Although your contact information will not be made public to the world via WHOIS when you use it, if a law enforcement agency requires us to provide them with your contact info, we will of course have to comply!

Just putting that out there!


 
© 1996-2006, DreamHost.com
Entries (RSS) and Comments (RSS). DreamHost Status
4:46 pm

Nimitz reboot.

Posted (April 28th, 2006 at 4:46 pm PST) by mir

My apologizes to users on Nimitz, I rebooted it once and it went haywire again before I could see what was going on so I had to reboot again. It’s doing just fine now but I wasn’t unfortunately able to determine the cause – I’ll be keeping a close eye on it though with a shell logged into the machine and top running so if it starts acting up again I should be able to catch it.


7:45 pm

Fairfax Failover

Posted (April 27th, 2006 at 7:45 pm PST) by Patrick

The ‘fairfax’ server’s hardware went kaput! We are failing it over to new hardware now and everything will be back up and running soon!


11:14 am

Testing New Web Log Rotation System

Posted (April 26th, 2006 at 11:14 am PST) by Dallas

Our current web log rotation system relies on our back-end automation system as well as steady access to our back-end database to run properly. It has been in place for many years and works fine most of the time but it is more complex than it needs to be and occasionally fails to work properly. That happened yesterday, for instance, and is the reason your logs may not have rotated. It worked properly today and everything is back up to date currently.

Rotating logs should not rely on other machines and we have developed a new version of the system that puts most of the work directly on the machine housing the logs, as it should be. The old system seemed like a good way to do it at the time, but it has not scaled well.

We are currently testing this new log rotation system on our hosting machine Orbital and will be widening the scope of the test to more machines if things go well. The first night of testing appears to have gone well.

If your hosting is on Orbital and you believe your web logs were not properly rotated last night (it would also prevent your Analog stats from running), let our support team know.


9:32 am

Central database issues

Posted (April 24th, 2006 at 9:32 am PST) by DreamHost

We’re experiencing central database issues which is affecting the performance of the control panel. The admins are currently looking into it. We’ll post an update as soon as we know further. We apologize for the inconvenience this may cause.

Please note: YOUR SITES, EMAIL AND DATABASES ARE NOT AFFECTED by this.


3:56 pm

Rails Upgrade, Part Deux

Posted (April 21st, 2006 at 3:56 pm PST) by Terri

We will slowly roll out the Ruby on Rails upgrade next week, starting Monday April 21. We will start with two machines first, and then upgrade the rest of the servers. Please freeze your Rails applications if you have not already.

To freeze your Rails application, all you have to do is run rake freeze_gems and it’s done! If you ever need to undo the freeze, you just need to do rake unfreeze_rails.

We promise that this upgrade will be a lot smoother than the last run!


3:44 am

Network File Server IP goof

Posted (April 21st, 2006 at 3:44 am PST) by DreamHost

It looks like one of our network file servers was missing an IP address from its network configuration. Initial testing/poking about with it wasn’t promising for a fix. Further futzing revealed that we may have been trying to assign it to the wrong physical network interface. It looks like our servers are picking up the mounts again without much prodding, so hopefully this is a permanent fix!

Our file server expert is going to look at it in the morning to double check my work however, because fixes at 3:45AM can have a way of undoing themselves.

(And after much futzing around, my first Blog post. Hopefully I set all the right options.)


4:43 pm

Digest Authentication enabled on Apache 2

Posted (April 19th, 2006 at 4:43 pm PST) by Dallas

We have updated our Apache 2 configuration so you can use Digest Authentication in addition to the more standard Basic Authentication. Most of our Apache servers are still Apache 1 and this change is only for Apache 2. Anyone who has added a subversion repository through our web panel is currently hosted on Apache 2 and can take advantage of this. If you would like your website to be on Apache 2, add a subversion repository to it. The act of adding the repository moves your website over. You can remove the repository afterwards if you like and your website will stay on Apache 2.

This configuration change will gradually roll out to all of our servers so if it doesn’t work when you first try it, give it a little time and try later.


12:14 pm

Clarifications on this site

Posted (April 17th, 2006 at 12:14 pm PST) by Dallas

Thank you all for the comments! We appreciate the praise for the design but the praise really should go to WordPress and the great selection of templates available. This is just an ‘off-the-shelf’ template customized a little bit.

We have also seen some confusion about how this website will be used and what you, as a reader, need to do to keep up with it.

First, this website fully replaces the previous status.dreamhost.com page. All previous issues have been resolved and have not been migrated into this interface. All future posts will go into this interface and only here. This is hosted off of our main network to ensure you will always have access to it. There are potential failures of our system that are beyond our control, such as the Los Angeles power outage 6 months ago, and building in fault tolerance is the best way to handle the unforseen.

Second, this new version of the website was partially spurred by customer requests for us to publish more information about changes we make to our servers, software, and network. We typically only send out official system announcements about major changes that are likely to affect a large pool of our customers. Also, we have only posted outages to the previous Emergency Status Page that would adversely affect your ability as a customer to access our primary method of contact (our announcement system). This website will now take over that Emergency Status Page role as well as provide a way for those of you interested in more information about system changes to hear about them. We expect a lot more information to be posted here than we have ever previously made available. It may take some time for us to fully integrate this new site into our procedures but that is the goal!

How does this thing work?

All posts will show up on this main page as well as in the main RSS feed. You do NOT need to click around the categories to see if there is something hidden under one of them. Also, the categories themselves are hierarchical and any ‘higher’ category automatically includes anything ‘below’ it. The hierarchy is visually demonstrated by the indenting of the category names and their positions.

The categories and their associated RSS feeds are provided as a convenience for those of you who do not want to see the information posted here in its entirety. For instance, you may want to see all of the ‘System Outages’ posts, but only the ‘Email Changes’ under the ‘System Changes’ category. To accomplish that you would subscribe to the ‘System Outages’ rss feed (which includes all of the categories below it!), and the ‘Email Changes’ rss feed. You could also just keep bookmarks of those category pages and check them regularly. The categories and associated rss feeds are set up in a hierarchy to provide you with as much flexibility in fine-tuning your information access as possible.

The current layout of categories is not set in stone. We may make changes in the future based on your feedback. The whole point of this website is to provide you with the information you want most as conveniently as possible.

If you have additional questions post them in the comments, and I will address them for you.


11:43 am

Earlier Network Outage Resolved

Posted (April 17th, 2006 at 11:43 am PST) by Dallas

This information was posted to the previous version of status.dreamhost.com but did not survive the transition to this new version. Sorry!

The network issue previously reported on April 12, 2006 at 5:49 PM PST was fully resolved at around 7pm PST that same evening. The problem stemmed from the addition of some new IP address blocks that went slightly awry. One block of our IP addresses was not routing properly for the duration of the outage. All other blocks were still working correctly. The particular problem had not come up before and now that we have experienced it we will take care to avoid it in the future.


3:27 pm

New DreamHost Status

Posted (April 14th, 2006 at 3:27 pm PST) by DreamHost

Welcome to the new and improved DreamHost off-network status page. This page will take the place of the previous status.dreamhost.com page as well as also host additional information about the technical goings ons behind the scenes at your favorite web host.


 
© 1996-2006, DreamHost.com
Entries (RSS) and Comments (RSS).